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This document includes three papers which address the 
special demands that supported employment places on vocational 
evaluation and assessment practices. An introductory chapter by Frank 
Rusch et al. discusses target populations of the Illinois Supported 
Employment Program, program evaluation, degree of program 
implementation, and costs and benefits of supp^**' employment. The 
first paper, titled "Vocational Evaluation and Eligibility for 
Rehabilitation Services" by Bruce Menchetti and Frank Rusch, 
describes the evolution of current vocational procedures and the 
utility of current practices in the development of effective 
supported employment programs, and recommends changes in evaluation 
and eligibility procedures. In "The Use of Standardized Assessment in 
Supported Employment," Lizanne DeStefano takes the position that 
existing assessment techniques contribute valuable information for 
decision making and documentation within the supported employment 
model. The article presents a number of standardized techniques, 
evaluates their strengths and weakness >s for use with severely 
handicapped individuals, and gives guidelines for selection and use 
of standardized tests in supported employment. Menchetti and Rusch, 
in "An Analysis of Select Psychometric Proparties of the Vocational 
Assessment and Curriculum Guide,** report on the development of a 
vocational and social skills rating scale that includes skills 
identified by employers, and examine the instrument's reliability and 
validity. (JDD) 
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The following pnnaptes guide our research related to the education and employment of youth and 
adults with specialized education, training, empioyment, and adjustment needs 



• Individuals have a basic nght to be educated and 
to work in the environment that least restricts their 
right to learn and interact with other students and 
persons who are not handicapped 

• individuals with varied abilities, srxial 
backgrounds, aptitudes, and learning styles must 
have equal access and opportLnity to engage in 
education and work, and life-long learning 

• Educational experiences must be planned, 
delivered, and evaluated based upon the unique 
abilities, social backgrounds, and learning styles of 
the individual 

• Agencies, organizations, and individuals from a 
broad array of disciplines and professional fields 
must offectively and systematically coordinate their 
efforts to meet individual education and 
employment needs 



• Individuals grow and mature throughout their lives 
requiring varying levels and types of eoucational 
and employment support 

• The capability of an individual to obtain and hold 
meaningful and productive employment is 
important to the individual's quality of life 

• Parents, advocates, and friends form a vitally 
important social network that is an instrumental 
aspect of education, transition to employment, and 
continuing employment 
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Preface 



Supported eraployment is rapidly taking its place as the 
preferred vocational service option for a large number of 
persons with handicaps. Federal and state legislative and 
fiscal commitment seem to secure that status. The special 
features of supported employment, with its emphasis on community 
employment and ongoing support, require the development of new 
technologies and practices in prooram planning, training, and 
program evaluation. Vocational evaluation has long been the 
means of obtaining information that can be used to guide these 
activities. In order to be maximally useful in supported 
employment efforts, vocational evaluation techniques must be 
sensitive to thos3 unique aspects of supported employment. 

In this volume of Supported Employment in Illinois we have 
chosen to include three papers that address the special demands 
that supported employment places on vocational evaluation and 
assessment practices. Although some themes remain common 
throughout the three papers, each approaches the topic in a 
slightly different way. 

In "Vocational Evaluation and Eligibility for Rehabilitation 
Services," Menchetti and Rusch attempt to build a case for 
changes in the traditional vocational evaluation procedures of 
the rehabilitation process. It is their contention that many of 
the empirically based tenets of supported employment have direct 
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implications for vocational evaluation. Given the focus on 
systematic training, supported employment evaluation efforts 
have to be designed to identify needed training resources and 
individual training objectives. In order to facilitate 
community job placement, vocational evaluation measures must be 
community referenced, that is, closely related to labor market 
needs. Finally, long-term follow-up services associated with 
supported employment will require evaluation techniques that are 
continuous to ensure that retraining can be provided and that 
employer satisfaction is monitored. 

To insure that evaluation information is functionally 
related to community work opportunities and the employment needs 
of individuals, the evaluation piocess must contain several 
components. These critical components are evaluating local 
labor markers, evaluating potential employees' vocational and 
related skill needs, developing individualized training plans, 
continuously monitoring program efforts, and evaluating employer 
satisfaction on a regular basis. Readers of tnis paper may want 
to examine the extent to which evaluation practices in their own 
agencies include each of the critical components. The 
recommendations and examples of a more functional approach to 
vocational evaluation of persons in supported employment 
programs can be used as templates to improve service delivery at 
the local level. 

DeStefano takes a different, yet complementary stance in 
"The Use of Standardized Assessment in Supported Employment." 
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Acknowledging that traditional work-sample systems, tests of 
academic skills, language batteries, and special-aptitude tests 
bear little relation to actual job requisites and have extremely 
limited validity for the population of persons involvea in 
supported employment, DeStefano advocates the use of situational 
assessment, ecological inventories, and criterion-referenced 
measures in supported employment settings. However, she goes on 
to cite instances in which standardized assessment information 
is also valuable in a supported employment setting and argues 
that it should not be excluded entirely from the vocationcii 
evaluation process. 

For example, intelligence test scores should not be used 
to exclude persons from service on the basis of perceived low 
ability, but they can be useful to summarize information 
regarding the types of persons participating in a program as an 
aid to administrators and planners of service. Adaptive 
behavior scores can further describe population characteristics, 
because they often include items that indicate community 
functioning. Adaptive behavior scores can also be used in a 
repeated-measures fashion co document the adaptive progress of 
an individual in supported employment or the overall influence 
of integration in areas other than employment, such as leisure 
and residence. This evidence can be used for program evaluation 
purposes to document changes in independent living as a result 
of participation in supported employment. In a similar manner, 
quality-of -lif e measures can be used to assess changes in 
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lifestyle satisfaction as a measure of program success. Both 
traditional and contemporary assessment techniques contribute 
valuable information for making decisions and documenting 
outcomes associated with supported employment. The task that 
service providers face is not whether to use one approach over 
the other but to devise an assessment package that adopts the 
best of both traditional and contemporary approaches. 
DeStefano's paper offers guidelines for doing so. 

Finally, in "An Analysis of Select Po/chometric Properties 
of the Vocational Assessment and Curriculum Guide," Menchetti 
and Rusch demonstrate that an instrument with psychometric 
properties associated with standardized instruments can be used 
as a measure of the social and vocational skills of persons with 
handicaps who are interested in competitive employment and can 
also provide information that can be used to develop training 
plans and instructional programs. The VACG is the first of a 
new wave of assessment instruments developed for use in 
supported employment. At this time it is the only instrument on 
the market that takes into consideration employers' expectations 
for their employees' performance on the job. It approaches the 
notion of content validity in terms of the local labor market 
and social norms. Service providers may want to consider the 
use of the VACG in their own supported employment programs. 

We hope that the papers included in this volume will provide 
useful ^.nformation, offer helpful suggestions, and stimulate 
thought among those of us interested in supported employment in 
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Illinois and throughout the narion. We welcome your cununenrs 
and suggestions and wish you the best of luck in your efforts. 

Lizanne DeStefano 
Frank Rusch 
July 1987 
Urbana, Illinois 
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Introduction to Supported Employment: 
Costs and Benefits 

Frank R. Rusch, John S. Trach, Debbie L. Winking, 
Jeffrey J. Tines, and Laird Heal 



Competitive employiTient is the normal and expected career 
path for persons who are nonhandicapped. The opportunity to be 
a part of the work force produces profitable personal and 
societal outcomes. Although competitive employment may not be a 
suitable option for everybody, it should be available so that 
all persons can, to the greatest extent possible, enjoy and 
engage in work that may result in individual or societal gains 
(Rusch, 1986). Across the country, supported employment 
programs are making competitive employment possible for a group 
of individuals for whom competrtive employment has not 
traditionally occurred or has been interrupted or intermittent 
as a result of their handicaps. The State of Illinois has been 
a leader in the movement to make supported employment a readily 
available outcome in the adult service delivery system. In this 
volume, we examine some salient characteristics of supported 
amployment in Illinois, including the target population, the 
degree of model program development, and the costs and benefits 
of supported employitient . 
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W ho Is Served in Illirois ? 

The 'lOis Supported Employment Prograun is designed to be 

an alter. .-.cive to day activity and sheltered employment 
programs. Owing to the interest in utilizing supported 
emplo^Tnent to extend services to individuals previously excluded 
from integrated work opportunities, it has been important for 
service provide. s collect data that explicate the range of 
participant characteristics. Table 1 presents some of the 
characteristics of the 514 individuals who have received 
supported employment services from the 30 adult service agencies 
throughout Illinois during two periods, February 1987 and June 
1987. The characteristics of persons served remain stable 
across time. As of June 1987, the mean age of these workers was 
approximately 30 years. Nearly 64% of the workers were male- 
The primary handicapping condition of most workers was mental 
retardation with full-scale IQ scores of the participants 
ranging from 17 to 97, with a mean score of 59.7. Secondary 
impairments such as cerebral palsy and physical handicaps have 
been identified for approximately 42% of the workers. 
Seventy-two percent of the workers are white; 20% are black. 
Hispanic and Asian workers are served in lesser numbers. The 
majority (52.9%) of the workers live in natural or adoptive 
homes; 19.2% of the participants live indepe'^dently or 
semi-independently ; just over 8% reside in an Intermediate Care 
Facility for Developmentally Disabled; and 11.6% live in a 
Community Residence Facility. This last figure represents a 
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Table 1 
SEP Worker characteristics 
(N = 514) 



Characteristic 


February' 1987 


June 1987 


Change 


Mean age (years) 


30 . 6 


30 . 6 


0 . 0 


Average full-scale IQ 


61 . 1 


59 . 7 


-1 . 4 


Sex (% of total) 








Male 


64 . 4 


63.6 


-0 . 8 


Female 


35 . 6 


35.7 


+0 . 1 


Not given 




0.7 




Ethnicity (% of total) 








White 


71.7 


72.0 


+0 . 3 


Black 


22.8 


20.0 


-2.8 


Asian 


1.2 


X.3 


-0.1 


Hispanic 


4.3 


5.4 


+1.1 


Unknown 




1.3 




Living arrangement (% of 








total) 








Natural/adoptive home 


53.4 


52.9 


-0.5 


Semi -independent living 


17.5 


19.2 


+1.7 


ICFDD 


8.4 


8.0 


-0.4 


CRF 


5.8 


11.6 


+ 5.8 


Other 


14.9 


8.3 


-6.6 
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100% increase from the earlier period in the number of persons 
served by supported employment who live in community residences. 

The majority of the workers have previously received some 
form of vocational training, ranging from high school vocational 
education courses to Developmental Training I services. In 
addition, 66.7% of the individuals have previously worked in 
sheltered facilities, and 43.7% of the individuals have had 
previous experience in competitive employment. 



Program Evaluation 



The Illinois Supported Employment Program (ISEP) based at the 
University of Illinois provides technical assistance to 
approximately 30 state-funded supported employment programs. In 
fact, ISEP is evaluating model program development and analyzing 
costs associated with supported employment. An important aspect 
of model program development is assessing the degree to which 
these model programs actually implement the characteristics of 
supported employment that have been associated with important 
outcomes, such as average hours worked per month, hourly wage, 
and employment benefits, both monetary and nonmonetary. 

ISEP also examines individual worker characteristics such as 
full-scale IQ scores, previous vocational training history, and 
residential living arrangements to monitor and evaluate the 
employment achievements of target employees. In addition, ISEP 
measures employment service characteristics such as the hours of 
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vocational skills training, assessment, and case management 
services delivered each month. 

Degree of Program Implementation 

In December 1985, field-based technical assistance was 
initiated using scheduled visits to eac^ supported employment 
model demonstration program to collect implementation data. All 
visits followed the same formula based on the evaluation 
instrument Degree of Implementation (DOI). During four rounds 
of visits, model programs (a) were introduced to the instrument 
and practiced scoring of the DOI (Round 1 - December 1985 to 
February 1986); (b) collected data on their programs (Round 2 - 
March 1986 to April 1986); (c) had the opportunity to react to 
the first year challenge of developing, implementing, and 
documenting progress in their efforts to establish supported 
employment in their respective communities (Round 3 - May 1986 to 
June 19&6). The fourth round of visits (December 1986 to 
February 1987) was designed to document the growth and/or 
stability of the efforts and to determine the validity and 
utility of the DOI as a standard for e^^aluation of supported 
emp?.oyment programs. 

The Degree of Implementation (DOI) Instrument * The DOI is an 
instrument bas^^d on the research literature related to national 
model demonstration development of supported employment programs 
(e.g., projects in Illinois, Washington, Vermont, and Virginia). 
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The intent of this instrument is to provide the Illinois 
Supported Employment Project (ISEP) with a standard to evaluate 
the implementation of the state supported employment initiative 
and a method for assessing the technical assistance needs of 
individual model programs. 

The DOI is designed to evaluate the process of developing and 
maintaining a supported employment model. It can be (and has 
been) used (a) to provide structure for beginning projects to 
establish supported employment programs by informing them of 
relevant activities identified through the literature, (b) to 
analyze the progress of the development of supported employment 
projects and to document the project's efforts in relationship to 
a specified time frame, (c) to investigate and identify possible 
variables that might facilitate program development, (d) to 
analyze the proposed model in relationship to actual documented 
services being provided, and (e) to investigate the relationship 
of the model to selected outcome variables (e.g., level of worker 
served, hourly wage, tenure). The instrument lists 28 steps or 
indicators that are categorized according to five components of 
supported employment programs: (a) Job Survey and Analysis, (b) 
Job Match, (c) Job Acquisition and Maintenance, (d) Conjunctive 
Job Services/Interagency Coordination, and (e) Job Fit. Using 
written documentation provided by the project, the evaluator 
scores the presence or absence of each indicator either as 0, 1, 
or 2, or NO (nonexistent), EMERGENT (present but incomplete), and 
YES (present and complete), respectively. Pre-established 
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written criteria determine the scoring of each indicator and 
serve as a manual for the administration of the instrument. The 
overall reliability obtained from the last set of ratings was 
.87 (Range = .75 to 1.00). 

Data collection result s. The results of efforts at 
implementing the proposed model of supported employment are 
presented in Table 2 and Figure 1. Level of implementation is 
expressed in quartiles. The first quartile (0 - 25%) is the 
lowest level of implementation and indicates the degree 
(percentage of DOI activities) to which a particular project has 
been implemented. The fourth quartile (76 - 100%) is the highest 
level of program implementation. Table 2 provides the 
percentages for each quartile in each round of data collection. 
The most current data demonstrate that 87% of the projects are in 
the top three quartiles of implementation and that the most 
growth has occurred in the second quartile, whereas the third and 
fourth have remained relatively constant. At this writing 13% of 
the projects have implemented 21 or more of the 28 supported 
employment activities (fourth quartile). The same percentage of 
projects implement 7 or less activities (first quartile), whereas 
47% of the projects implemented between 8 to 14 activities 
(second quartile) and 27% of the projects implemented 15 to 20 
activities (third quartile). Data on overall implementation over 
the six-month period indicate that the number of programs 
implementing more than 50% of the activities is increasing. 
Figure 1 shows the steady decline in the first quartile and a 
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Table 2 

Percentages of Overall Scores by Quartile 
Degree of Implementation 



Quartile 



Number 

of DOI 
Activities 
Implemented 



Percentage of Projects at Each 
Level of Implementation 



March to 
April 
1986 



May to 
June 
1986 



December to 
February 
1987 



1. 0-25% (0-7) 42 
implementation 

2. 26-50% (8-14) 21 
implementation 

3. 51-75% (15-20) 27 
implementation 

4. 76-100% (21-28) 9 
implementation 



30 
33 
18 
18 



13 



47 



27 



13 



redistribution into the remaining three quartiles over the three 
rounds of DOI data collection. 

The items implemented most frequently are those activities 
that sarvey the community (#1), task analyze potential jobs 
(#6), identify requisite skills (#8), assess and observe 
vocational skills (#12 and #14). and reassess through 
observation the client's maintenance of vocational skills (#26). 

Although, the items cited in the previous paragraph 
represent the core of activities that most projects are 
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Degree of Implementation Summary Data 



Percent of Implementation 




m 4th Qrtile 
3rd Qrtile 
2ni Qrtile 
1st Qrtile 

First Round Second Round Third Round 
Round of Data Collection 

implementing, they do not necessarily indicate all of the 
activities that are associated with successful supported 
employment prograir.s. There are some important activities that 
many projects are not implementing. For example, the 
identification and assessment of social skills are implemented 
at a significantly lower rate than items related to vocational 
aspects of employment. Ironically, research literature 
indicates that persons with disabilities lose their jobs most 
often because of social skill deficits (Greenspan & Shoaltz, 
1981). It would seem important, therefore, for projects to 
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conduct social skills assessment and intervention activities. 
It is encouraging that more social skills assessments occurred 
in the last round than in previous rounds, it could be that as 
projects gain experience, the need for attention to social 
ski] Is becomes evident. 

It is also troubling that there is a lack of systematic 
training, data collection, and withdrawal. Only 37% implement 
this aspect (Item #16) of the Job Acquisition and Maintenance 
component. Because effective systematic training and data 
collection strategies are critical to successful supported 
employment, staff selection and development activities should 
seek to improve procedures in this area. The low-level 
implementation of Job Acquisition and Maintenance activities may 
be attributed to the level of worker being served by the 
initiative. There is some indication that there is an inverse 
relationship between the level of worker functioning and scores 
on the Job Acquisition and Maintenance component of the DOI . 
This has been interpreted to mean that workers with higher IQs 
do not require as much attention to training as workers with 
lower IQs (Trach & Rusch, 1987). 

Summary. The DOI data collection indicates that there is a 
positive trend to increase the implementation of supported 
employment activities in Illinois since June 1985. Job Survey 
and Analysis and Job Match are the most widely implemented 
components; the remaining three components Job Acquisition 
and Maintenance, Conjunctive Job Services/Interagency 
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Coordination, and Job Fit — are implemented at consistently low 
levels. Some possible reasons for nonimplementation of DOI 
activities include: (a) lack of documentation, (b) inability to 
implement because of staff resources or lack of technology, (c) 
staff resistance to change, (d) level of worker functioning, and 
(e) philosophical differences. Model program development will 
continue to be evaluated during the next year. Volume 3 of 
Supported Employmen t in Illinois will report upon DOI data 
collected during the fourth quarter (May-June, 1987). 

Costs and Benefits of Supported Employment 

A comprehensive evaluation of the statewide supported 
employment initiative requires the explicit identification of 
the costs and benefits associated with it. In response to the 
need to define the supported employment model in Illinois 
economically, an "accounting approach" (Thornton, 1985) was used 
to estimate the benefits and costs of the statewide effort from 
the 30 model programs. By using an "accounting model," a 
detailed itemization of each expenditure and amount of revenue 
for each supported employment project was determined from 
several perspectives. For this evaluation of the supported 
employment initiative, two different perspectives need to be 
examined. The first perspective considered was that of the 
individual service recipient, while the other was that of the 
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taxpayer or government agency. 

From the perspective of the individual service recipient 
(worker), the potential monetary benefits include the gross 
income earned through the production of valued goods and 
services. In contrast, fi^m the taxpayer's perspective a 
benefit of the initiative is a decrease in government subsidy 
that results from an individual's decreased Supplemental 
Security Income payments. At the same time, the costs 
associated with supported employment from a taxpayer perspective 
or government agency are the operational costs that governmental 
agencies assume by funding the provision of the services. In 
addition, a cost of supported employment from the individual 
worker's perspective is the increased taxes withheld from the 
additional earnings that result from employment. 

Cost-benefit research on employment services for individuals 
with handicaps has been limited. However, there are two studies 
that should be examined. Hill et al. (1987) used an accounting 
model (Thornton, 1984) in their cost-benefit analysis of a 
supported competitive project in Virginia. They determined that 
supported competitive employment for individuals with mental 
retardation cost the government agency/taxpayer $8,717 per 
consumer while this same group realized a benefit of $15,282 per 
consumer during a 94-month period. In other words. Hill et al. 
determined that for each $1.00 expended by the government/ 
taxpayer for supported competitive employment, $1.87 was 
accumulated in benefits. From the consumer's perspective, 
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the benefit also outweighed the cost of the program with a $1.97 
accumulated benefit for each $1.00 expenditure. 

A second study by Schneider, Martin, Rusch, and Geske (1981) 
examined the costs associated with training 23 individuals with 
mental retardation in a transitional employment progr^.m to 
become food service laborers at a university-based cafeteria. 
This study compared the benefits (i.e., earnings) of both 
extended eniployment programs (e.g., sheltered employment) and 
transitional employment programs (e.g., competitive employment 
food service training). The findings revealea that during the 
third year of the program the total earnings exceeded the costs 
associated with placement. Additionally, the study revealed 
that by the end of the fifth year, earnings associated with the 
employed workers exceeded the costs required to support the 
individual service recipient. 

As a precursor to evaluating the statewide initiative with a 
cost-benefit analysis, a summary was completed of the reported 
financial information on individuals who have participated in 
the initiative. This information was used to answer several 
questions dealing with the financial status of participants in 
supported employment, such as: 

1. What are individuals with disabilities earning 
both monthly and yearly as participants in 
supported employment? 

2. How many hours have the participants in the 
supported employment initiative been working 
each month? 
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3, How much monciy has been withheld in taxes from 
the earnings of individuals in supported 
employment? 

In response to these questions. Table 3 summarizes the data 
for two different periods of time, August-December 1986 and 
January-March 1987. During the period from January to March 
1987 the average hourly pay was $3.21- The average number of 
hours worked was 85.5 hours per month. The average gross pay 
per month was $276.47 and the average amount of taxes withheld 
was $34.76 per worker en a monthly basis. The average number of 
hours worked per month decreased from the first reporting period 
to the second, accompanied oy resulting dec^'eases in gross wages 
and taxes withheld. 

Table 3 

Cost-benefit Analysis of the Supported Employment Initiative 



Variable August-December 1986 January-March, 1987 



Mean 


hourly wage 


$ 3.04 


$ 3.21 


Mean 


hours worked/month 


94.91 


85.50 


Mean 


gross wage/month 


308.58 


276.47 


Mean 


taxes withheld/month 


45.72 


34.76 
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As some of these more general questions related to the 
financial status of the participants in supported employment are 
answered, more complex ones arise. Given the accounting 
framework provided by the work of Thornton (1985), most complex 
questions related to the costs and benefits of supported 
employment can be addressed* The following questions will serve 
to direct the next round of cost-benefit analyses conducted by 
^ Illinois Supported Employment Program (ISEP) • 

1. What are some of the non-salary expenditures 
associated with supported employment efforts? 

2. How much of an agency's management support 
does a supported employment program demand? 

3. How do the costs associated with alternate day 
services compare to those of supported employment 
services? 

4. How do the net expenditures and benefits of 
supported employment statewide compare? 

Summary 

Since the start of the supported employment initiative in 
Illinois, much has been accomplished toward making the benefits 
of competitive employment available to persons with handicaps. 
Program implementation data indicate that model programs are 
carrying out greater numbers of supported employment activities 
than ever before. Although improvement in assessment and 
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training is still neede*^ • program evaluation data indicate that 
model programs are becoming proficient in job development, job 
analysis, ana job matching activities. More than 500 persons 
are currently working in the supported employment model programs 
around the state. These persons work an average of 86 hours per 
month at a mean hourly wage of $3.21^ earning an average of $277 
gross pay per month. The supported employm.ent program is a new 
one, and the chanllenges facing prospective employees with 
handicaps^ adult service providers, and educators are numerous 
and complex. However, the initial evaluat'on of the initiative 
gives clear evidence of its success and of progress made toward 
improving the quality of life for persons with handicaps. 
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Vocational Evaluation and Eligibility for 
Rehabilitation Services* 

Bruce M. Menchetti** and Frank R. Rusch 



The rehabilitation process, consisting of intake, referral, 
evaluation, individualized planning, treatment, training, 
placement, and closure, is designed both to determine the 
eligibility for service of individuals with handicaps and to 
provide appropriate service to those who are eligible. Two 
criteria are used by state vocational rehabilitation personnel 
to determine an individual's eligibility to receive services: 
(a) the applicant must have a physical or mental disability that 
interferes with his or her employment, and (b) a reasonable 
possibility must exist that rehabilitative services will result 
in gainful employment • 

Vocational evaluation is pivotal to the rehabilitation 
process. In conjunction with intake and referral information. 



* Reprinted with permission from P. Wehman & S. Moon (in 
press ) . Vocational rehab^ xxtation and supported 
employment . Baltimore: Paul H. Brookes. 
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evaluation data are used to identify disabilities and to 
determine whether or not an individual has a reasonable chance 
of getting and keeping a job* As a result, vocational 
evaluation has played a key role in determining eligibility for 
rehabilitation services and thus has become the component of the 
rehabilitation process with the most direct impact upon persons 
seeking employment services. 

For some applicants of vocational rehabilitation services, 
the connection between evaluation and eligibility is 
straightforward. For instance, medical evaluations are 
frequently useful for identifying the medical or psychiatric 
treatment needed to reduce an individual's disability. Once the 
applicant has been determined t^^ be eligible and appropriate 
services have been identified, the rehabilitation process can 
ontirue toward job placement and eventual closure* For many 
vocational rehabilitation applicants, however, evalu^^^ion 
practices do not always result in such precise eligibility and 
service outcomes. For individuals with severe disabilities such 
as mental retardation, cerebral palsy, and autism, vocational 
evaluation often results in ineligioility. 

Given the supported employment mandate of the Rehabilitation 
Act amendments (P.L. 99-506), professionals in vocational 
rehabilitation, developmental disabilities, mental health and 
retardation, and special education will have to re-examine their 
vocational evaluation practices. Evaluation procedures that do 
not provide meaningful information related to planning effective 
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programs for persons with severe handicaps will be of no help to 
these practitioners as they face the challenge of supported 
employment. When testing procedures do not result in 
training-related information, evaluation becomes nonfunctional 
in the larger rehabilitation process. 

This chapter has three purposes: to describe the evolution 
of current evaluation procedures, to address the utility of 
current practices in the development of effective supported 
employment programs, and to recommend changes in evaluation and 
eligibility procedures. It is the au\:hors' intent to provide, 
the reader with suggestions that will be helpful in planning 
vocational evaluation activities that are closely related to 
supported employment training. We believe that vocation^^l 
evaluation should remain an integral and useful part of the 
rehabilitation process. 

Evolution of Current Vocational Evaluation Procedures 

Service providers can choose from a wide variety of 
evaluation procedures to identify the employment training needs 
of persons with disabilities, including measures of general 
intelligence, educational achievement, motor dexterity, 
mechanical aptitude, occupational interest, personality traits, 
and work habits. Many of these procedures have been adapted 
from methods developed by early researchers in vocational 
evaluation. Current practice, therefore, cannot be adequately 
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understood without examining the history of vocational 
evaluation. 

Standardized intelligence, achievement, and aptitude tests 
have been used for vocational evaluation since World War I. It 
v/as not until World War II, however, that these measures gained 
widespread use, when approximately 14 million men underwent some 
form of achievement or aptitude testing to evaluate their 
suitability for various military jobs. The work of Robert L. 
Thorndike during World War II shaped the field of vocational 
evaluation, and his influence is still felt today. 

Thorndike, who was the psychologist largely responsible for 
the development and administration of the Aviation Psychology 
Program of the Army Air Force, defined the goal of personnel 
testing as "selecting certain individuals from among the 
applicants for a job, or determining for which of two or more 
possible job categories a particular individual shall be 
assigned" (Thorndike, 1949, p. 4). This form of evaluation 
challenged the personnel psychologist to derive "insights and 
hypotheses as to the psychological functions requiied for 
success on the job" (p. 12). To accomplish this Thorndike 
suggested a rigorous method consisting of job analysis, 
selection and invention of testing procedures, preliminary 
tryout and refinement of instruments, validation of test 
procedures, combination of tests into a battery, and finally, 
systematic administration of the testing program. Thorndike 's 
approach to vocational evaluation was analytical, objective, and 
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empirical. He wrote, "The feature that distinguishes ieputable 
work in personnel selection from that of the mass of self-styled 
'psychologists/ 'personnel experts,' and other quacks is that 
the reputable worker in the field is continuously concerned with 
testing, reverifying, and improving the adequacy cf his 
procedures" (p. 2). Some of the psychological functions that 
Thorndike and his colleagues identified as prerequisites for 
successful Army Air Force pilots were an understanding of 
mechanical principles, knowledge of general information, complex 
coordination, instrument comprehension, and arithmetic 
reasoning. 

The approach to vocational evaluation developed by 
Thorndike was highly empirical. The procedures he used to 
develop, select, validate, and combine tests became known as the 
process of standardization (Neff, 1966). Because many of the 
characteristics Thorndike measured were related to both general 
and specialized psychological functions, Neff (1966) labeled 
this evaluation model the mental testing approach. 

The Mental Testing Approach in the Private Sector 

The mental testing approach to vocational evaluation 
developed and refined by Thorndike for the military was 
eventually adopted by business and industry, where prospective 
employees were evaluated with batteries of aptitude and 
achievement tests. 

Test authors like George K. Bennett developed instruments 
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designed to measure general vocational aptitudes • The Test of 
Mechanical Comprehension^ Form AA (Bennett, 1947) became one of 
the most widely used instruments for testing job applicants. 
The major purpose of this testing was to measure the applicant's 
ability to perceive and understand physical laws and practical 
mechanical relationships. Bennett claimed that this aptitude 
was important for a wide variety of jobs, with this assertion, 
the field of vocational evaluation which had originated to 
select individuals for highly specialized jobs such as piloting 
aircraft, became oriented toward the prediction of general 
vocational success . 

The vocational evaluation model which utilized standardized 
tests for predicting general vocational ability gained strength 
in the post^World War II era. Business and industry utilized a 
wide range of evaluation instruments to screen prospective 
employees for a variety of jobs. Many of these instruments, 
however, were a compilation of techniques iis^d by the military 
to test ability far specialized jobs (Cronbach, 1960). For 
example, the Flanagan Aptitude Classification Tests were a 
battery of 21 tests, suggested by Air Force studies, that 
included measures of scale reading, carving skill, and tapping 
ability. The Guilf ord-Zimmerman Aptitude Survey, published in 
1947, also contained measures found useful in Air Force 
classification (Guilford, 1947). Perhaps owing to its adoption 
by business and industry, the mental testing approach became the 
preeminent model of vocational evaluation. When education anc 
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rehabilitation professionals became involved in vocational 
evaluation, the mental testing approach was the methodology they 
selected. 

The Mental Testing Approach in Education and Rehabilitation 

The mental testing approach has been characterized as 
standardized testing to predict general vocational ability 
(Cobb^ 1972; Gold, 1973; Halpern, Lehman, Irviii & Heiry, 1982; 
Neff, 1966). The mental testing model with its prediction 
orientation has been applied to the vocational evaluation of 
individuals in educational and rehabilitation settings, and many 
of the evaluation techniques in use throughout the country 
reflect this approach. 

One of the first instruments developed for use in 
educational settings was the Differential Aptitude Tes ts (DAT) 
(Bennett, Seashore, & Wesman, 1947). Linn (1978) has called the 
DAT the Cadillac of multiple aptitude batteries. The DAT 9 
battery consists of eight tests, including verbal reasoning, 
numerical ability, abstract reasoning, clerical speed and 
accuracy, mechanical reasoning, space relations, spelling, and 
language usage. The tests that constitute the DAT were designed 
to facilitate the work of high school vocational guidance 
counselors as they advise students who are making career 
choices. Several reviewers have suggested that the DAT is one 
of the most thoroughly validated instruments of its kind 
(Cronbach, 1960; Linn, 1978; Kastie, 1976). Although disabled 
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subjects were not included in the DAT standardization sample, 
the instrument is characteristic of the mental testing approach 
to vocational evaluation. In fact, the evaluation methodology 
and prediction orientation used by the DAT have been 
incorporated into many current systems used by education axid 
rehabilitation professionals, including both the multi-aptitude 
batteries and the popular work sample systems used in many 
vocational evaluation prograuns. 

The U.S. Employment Service has developed the Nonreadinq 
Aptitude Test Battery (NATB) for vocational evaluation of 
"educationally deficient" individuals. The NATB is a nonreading 
version of the most widely used multi-aptitude battery, the 
U.S.E.S. General Aptitude Test Battery (GATE) (Borgen, 1983). 
The NATB subtests correspond very closely to those of the DAT* 
NATB subtests include verbal ability, numerical ability, manual 
dexterity, clerical perception, form percept-ion, spatial 
perception, general learning ability, motor coordination, and 
finger dexterity. It is interesting to note the similarities 
between the NATB, revised in 1981, and the DAT, which was 
developed in 1947. In fact, even some of the more work^-oriented 
vocational evaluation systems still closely resemble the DAT. 

The Valpar Component Work Sample System comprises 16 
subtests (Botterbusch, 1980), iTiany of which resemble subtests of 
the DAT. VALPAR subtests include independent problem solving, 
numerical sorting, money handling, clerical comprehension and 
aptitude, simulated assembly, size discrimination, electrical 
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circuitry and print reading, small tools, upper extremity range 
of body motion, multilevel sorting, whole range of body motion, 
trilevel measurement, eye-hand-foot coordination, soldering and 
inspection, integrated peer perfornance^ and drafting. Many of 
the VALPAR subtests focus on general work aptitudes, which is a 
defining feature of the mental testing approach to vocational 
evaluation. 

Two extremely popular and widely used vocational evaluation 
techniques, the multi-aptitude test battery (e.g., NATB) and the 
work sample system (e.g., VALPAR) are directly related to the 
mental testing approach. Some similarities between current 
evaluation techniques and the traditional mental testing 
approach include: (a) a focus on general work abilities that are 
presumably elements of successful vocational adjustment, (b) a 
prediction orientation and purpose, and (c) a heavy reliance on 
standardized instruments. The aptitude, achievement, ^nd work 
sample methods have found their way to the forefront of 
education and rehabilitation evaluation programs (Brolin, 1982; 
Phelps & Mccarty, 1984). Proponents of the approach have 
suggested that the behaviors measured with aptitude and 
achievement batteries may be related to an individual's 
employability. For example, Brolin (1982) nas said, "Academic 
skills in the areas of reading, writing, and mathematics play an 
important role in the determination of vocational potential" (p. 
91). The capability of mental testing techniques, most of which 
were originally designed to screen nonhandicapped persons for 
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highly specific jobs, to predict the general employment 
potential of persons with handicaps has been questioned 
repeatedly (Cobb, 1972; Gold, 1973, Menchetti, Rusch, & Owens, 
1983; Schalock & Karan, 1979; Wolf ensberger , 1967). Given these 
questions, the utility of the mental testing approach in 
vocational evaluation must be examined critically by education 
and rehabilitation professionals. 

Utility of Current Vocational Evaluation Practices 

There are several reasons why the mental testing approach 
has obtained widespread acceptance in the vocational evaluation 
of individuals with disabilities. Aptitude batteries, 
achievement tests, and work sample systems have been marketed 
aggressively by publishers. Many of the current vocational 
evaluation instruments have reported elaborate standardization 
and validation data (Field, Sink, & Cook, 1978; Flenniken, 1975; 
Hull & Halloran, 1975; Jones & Lassiter, 1977). These reports, 
however, have not served to persuade the critics of the mental 
testing approach or similar instruments used to evaluate persons 
with handicaps. 

Neff (1966) has pointed out that although the mental testing 
approach "seems like a triumph of empirical logic, one may be 
almost astonished to discover that even the most impeccably 
developed tests have respectably high reliabilities but 
disappointingly low predictive validity" (p. 55). Other 
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researchers have pointed out that many of the standardized 
measures used for vocational evaluation of persons with 
handicaps have not been sufficiently validated for this purpose 
(Gold, 1973; Wolf ensberger , 1967). Finally, some critics have 
suggested that the purpose of many vocational evaluation 
programs, namely the prediction of general employability , may be 
unrelated to the more relevant goal of identifying the specific 
training needs of persons with handicaps (Cobb, 1972; Halpern et 
al., 1982; Menchetti et al., 1983; Schalock & Karan, 1979). In 
order to judge the utility of current vocational evaluation 
practice for supported employment, one must excimine these 
criticisms . 



Criticisms of Current Evaluation Practices 

Many professionals have pointed out that there has been a 
lack of empirical evidence showing that scores on instruments 
currently used in vocational evaluation programs are related 
significantly to the employability of persons with handicaps 
(Browning & Irvin, 1981; Cobb, 1972; Gold, 1973; Menchetti et 
al., 1983; Schalock & Karan, 1979: Wolfensberger , 1967). Some 
of these individuals have criticized the validation research, 
whereas others have questioned the evaluation methodology and 
purpose. 

Twenty years ago, Wolfensberger (1967) criticized the 
validation research for several reasons, including poor 
methodology, lack of cross-validation studies, failure to 
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analyze training variables, and the assumption that the criteria 
defining successful employment were the same for all persons. 
Wolf ensberger ' s criticisms emphasized that training variables 
play a critical role in assessing the employability of persons 
with handicaps. These variables are typically not taken into 
account by current evaluation techniques, which suggested to 
many in the field that a vocational evaluation approach that 
predominantly measured general abilities, achievement, 
aptitudes, and other manifestations of prior learning was 
nonfunctional. This belief led to a movement to shift the 
purpose and orientation of vocational evaluation for persons 
with handicaps. 

In an important work entitled. The Forecast of Fulfillment , 
Cobb (1972) stated that there are two basic orientations to 
vocational evaluation, the prediction orientation and the 
counseling ^rientation. A major goal of evaluation approaches 
with a prediction orientation is the measurement of variables 
that presumably forecast future emploj^.aent potential. 
Approaches with a counseling orientation attempt to measure 
variables that can be used to identify the specific training 
needs of persons with handicaps. Techniques of evaluation with 
a counseling orieucation can be used for the curriculujm 
development and instructional planning needed to improve 
vocational training opportunities for individuals with 
handicaps. Many professionals he. e suggested that Cobb's 
counseling orientation is the only relevant vocational 
evaluatio approach . 
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Gold (1973) has criticized evaluation techniques designed 
prim .xly to predict a person's potential for employment, 
suggesting that the validity of instruments with a prediction 
orientation was statistically significant in a research context 
but was lacking in practical applicabixlty C^old was referring 
to the numerous studies that correlated scores on newlv 
developed aptitude batteries and work samples with measures on 
other, more established instruments (such as Distefano, Ellis, & 
Sloan, 1958; Tobias, I960; Wagner & Hawver, 1965). 

Gold pointed to three serious problems with this approacn to 
validation. First, many of the criteria measures, that is, the 
scores on the more established tests, had not established their 
own validity and reliability with handicapped populations. Gold 
also pointed out that any validation study that correlated 
scores on a newly developed test with current scores on similar, 
but more established *-ests was investigating concurrent 
validity. Cronbach {I960) suggested that concurrent validation 
data have limited -Generality when tests are used to make 
predictions regaroxng potential performance. Finallv, Gold 
re-emphasized that vocational evaluation with a prediction 
orientation has little relevance to the training needs o^. 
individuals with handicros. For example, determining that a 
person with mental retardation has a low IQ, or scores low on a 
general aptitude battery. Gold argued, reveals nothing about how 
to plan an appropriate course of vocational training for that 
individual. In fact, the poor performance on aptitude tests by 
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persons with handicaps is often used to justify their exclusion 
frora the very training they require. Like Wolf ensberger (1967), 
Gold (1973) called for a shift in the focus of vocational 
evaluation away from a prediction orientation and toward a 
training facilitation orientation. 

The professional call for a change in the focus of 
evaluation continued. Schalock and Karan (1979) called for an 
approach that emphasized a close, interactive relationship 
between evaluation and training activities, an approach they 
termed edumetric because its focus is on the measurement of 
education and training needs. Kalpern, Lehman, Irvin, and Heiry 
(1982) differentiated between traditional and contemporary 
evaluation approaches and suggested that traditional measures 
are useful when the goal is to identify prior learning such as 
aptitudes, interests, and traits. Halpern et al. (1982) pointed 
out, however, that when one is evaluating the vocational nee^s 
of persons with handicaps, traditional information is often 
redundant. A contemporary evaluation approach which meas'!^es 
applied work performance and social behavior in the context in 
^-'hich such performance is expected, would better facilitate 
identification of an individual's training needs and increase 
his or her access to training opportunities. Halpern et al. 
(1982) suggested the emergence of contemporary evaluation 
techniques as the most functional apprc^ach to vocational 
evaluation. Mos*. recently, Kokaska and Brolin (1985) stated 
that "norm-ref ert*xced tests that compare a student's performance 
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against the norm are not the most appropriate method of 
assessment to ascertain the individual's competence. In our 
opinion, a type of criterion-referenced ^cst that assesses the 
student's mastery of competence in specific areas is more 
useful" (p. 281) • 

Supported Employment; New Challerqes for Vocational Evaluation 

General agreement is emerging among professionals that 
vocational evaluation services for persons with handicaps should 
emphasize the identification of individual training needs over 
measurement of prior learning (e.g., achievement, general 
aptitudes, work habits) . Evaluation techniques that focus on 
prior learning highlight the limitations of the individual. 
Proponents of supported employment have collected data that 
suggest the difficulties many persons with severe handicaps 
experience when seeking to be competitively employed cannot be 
solely attributed to their limitations (Baces, 1986; Bellamy, 
Rhodes, Bourbeau, & Mank, 1986; Rusch & Mithaug, 1980; 
Vogelsberg, 1984; Wehman, 1981). Instead, these professionals 
have pointed out that the employment problems faced by persons 
with severe handicaps are, in part, a result of ineffective 
services and the larger societal employment context. In fact, 
the concept of supported employment is based upon research 
findings that corroborate this position. First, researchers 
have pointed to national studies of the outcomes attained by 
sheltered work programs, which serve the majority of persons 
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with handicaps (General Accounting Office, 1980; Greenleigh 
Associates, 1975; U.S. Department of Labor, 1977, 1979; 
Whitehead, 1981). These studies have shown that sheltered 
services have been largely ineffective in providing reasonable 
wages for their worker? , in moving them to higher levels of 
productivity, and ir placing them into the competitive labor 
force. Second, supported employment proponents have indicated 
that there is conclusive evidence of the productive capacity of 
individuals with severe handicaps. Using behavioral training 
techniques and a systerudtic approach to working with community 
employers, researchei. ^ have demonstrated that persons with 
handicaps can learn the skills needed to earn wages 
significantly above the sheltered workshop average (Rusch, 
1986). Based on these data, supported employment programs have 
prolifer ed. All of these programs assume that persons with 
har^-iicc n become productive mt nbers of our society. 

Many of the empirically based tenets of supported employment 
have had a direct impact upon vocational evaluation. Given the 
focus on systematic training, supported employment evaluation 
efforts have been designed to identify needed training resources 
and individual training objectives. Furthermore, with the 
emphasjs on community employment alternatives, vocational 
evaluators have focused their measures toward coirjnuaity 
expectations as these evaluations bear a close relationship to 
the needs of local labor markets (Pancsofar, 1986). Finally, 
vocational evaluation efforts have facilitated the data -based 
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instructional decisions used in behavioral training programs. 
The relationship of these supported employnient tenets to 
vocational evaluation are sununarized in Table 1, 

Some of the guiding tenets of supported employment have 
begun to be incorporated into vocational evaluation policy. The 
Vocational Evaluation and Work Adjustment Association (VEWAA) 
has suggested that vocational evaluation must benefit both the 
service provider and the client by providing information that 
facilitates the development of a plan of action (Schneck, 
1981). Recent legislation (e.g., PL 99-506 and the Carl D. 
Perkins Vocational Education Act) has placed demands on local 
rehabilitation and education agencies to condi ,t evaluation that 
results in both identification of the vocational reeds of 
persons with handicaps and increased access to needed services. 
Kokaska and Brolin (1985) have suggested that the only 
reasonable purpose for vocational evaluation is to facilitate 
the career development of the individual. Accordina to these 
authors, the major function of evaluation is that "the 
individual's strengths and weaknesses can be discerned so that 
the lEPs and the Individualized Written Rehabilitation Plans 
(IWRPs) can be planned, and individual and group progress can be 
monitored" (p. 281). The remainder of this chapter addresses 
the need for evaluation techniques to meet the challenges 
presented by supported employment. 
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Table 1 

Tenets of the Supported Employment Movement 
Related to Vocational Evaluation 



- The purpose of evaluation is to facilitate the 
identification of individual training needs and 
resources , 



Measures should be community referenced and 
interpretation of performance must assist 
instructional decision making. 

- Successful vocational adjustment is related to 
training variables and the employment context. 
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Evaluation in Supported Employment 

For many years, the multi-c.ptitude battery and work sample 
techniques have dominated the field of vocational evaluation. 
Although these techniques have been used to measure a variety of 
general aptitudes and skills, the relationship of these 
variables to the successful vocational adjustment of individuals 
with handicaps has not been established empirically. In fact, 
recent research has suggested that productivity is more closely 
associated with service provision. Services, including 
systematic instruction, frequent contact with employers, and the 
availability of long-term support, nave defined supported 
employment. Education and rehabilitation professionals 
interested in developing supported employment programs must 
adapt their current evaluation and eligibility procedures to the 
outcome-oriented approach of supported employment. 

Certain features of r.upported employment will require the 
use of unique vocational evaluation methods. Given the 
behavioral orientation of supported employment, evaluation 
efforts must result in information that can be used to plan 
individual training programs, suggest alternative training 
approaches, and determine whether outcomes are within an 
acceptable range of performance. The community employment focus 
of supported employment will also have direct implications for 
vocational evaluation. In order to facilitate community job 
placement, vocational evaluation measures must be community 
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referenced; that is, the instruments should be closely related 
to local labor market needs. Finally, the long-term follow-up 
services associated with supported employment will require 
evaluation techniques that are continuous, in order to ensure 
that retraining can be provided when needed and that employer 
satisfaction is documented and monitored. 

Ecological Analysis 

One evaluation technique that has been very useful to 
supported employment programs is ecological analysis. Applied 
to vocational evaluation, ecological analysis is the 
identif icatiorx and measurement of actual skills required for 
employment. Wehman, Renzaglia, and Bates (1985) have defined 
ecological analysis as a systematic approach to identifying 
skills that have a high priority for a person to learn. In 
employment settings, ecological analysis has also been called 
job analysis (Rusch & Mithaug, 1980; Schutz & Rusch, 1982). In 
supported employment, job analysis has been used as a strategy 
for conducting an empirical analysis of the employment ecology. 

One example of an empirical approach to identification of 
high priority employment skills is the Job Skills Inventory 
(Belmore & Brown, 1978). The Job Skills Inventory analyzes the 
employment ecology with a three-step strategy or process. 
First, the general vocational and social skill requirements of a 
specific job are identified. Second, workers employed in the 
target job are directly observed and each previously identified 
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skill is broken down into its component behaviors, a technique 
known as task analysis. Finally, critical factors such as an 
individual's transportation and independent living skills are 
identified. These factors are important to long-term job 
success and must be evaluated so that skill training and 
supportive services can be provided when necessary. 

The information obtained with the Job Skills Inventory is 
particularly relevant to supported employment. The work, 
social, and community skills identified through the inventory 
process can be easily translated into training objectives. 
Using behavioral observation techniques and evaluation of 
individual performance, a method for determining intervention 
effectiveness and planning needed program adjustments can be 
designed. Task analysis data collected during the inventory 
process can be used to establish normal levels and rates of 
worker productivity. This information is important because it 
allows evaluators and trainers to judge when a target employee's 
work performance is acceptable. The job inventory strategy 
assures that skills are referenced to the local labor market and 
have a high probability of being valued by community employers. 
Finally, the job inventory process can be used to develop 
evaluation instruments such as skill checklists for continuous 
measurement of worker performance. The inventory process 
represents a promising technique for supported employment. 
Figure 1 depicts the job inventory process, culminating in the 
development of an individualized training plan. 
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Another evaluation instrument based upon an ecological 
analysis of employment opportunities is the Vocational 
Assessment and Curriculum Guide (VACG) (Rusch, Schutz, Mithaug, 
Stewart, & Mar, 1982). The VACG includes a variety of general 
work and social skills based on an empirical analysis of job 
demands. Rusch, Schutz, and Agran (1982) surveyed employers in 
service and light industries to determine the skill demands of 
their entry-level jobs. The results of this survey provided the 
item pool for the VACG. 

The VACG has been designed as a behavior-rating scale that 
provides a measure of the vocational and social skills of 
persons with handicaps. The VACG comprises ten domains: 
attendance/ endurance , independence , production , learning , 
behavior, communication, social skills, grooming/eating, 
reading/ writing, and math. There are 66 items on Lhe VACG, 
each beginning with the phrase, "Does the worker," followed by a 
description of the behavior being assessed. Several possible 
responses are provided that indicate levels of performance 
displayed by the worker. Raters are instructed to select the 
phrase that best describes the individual's current level of 
functioning. The VADG was designed to be used by classroom 
teachers, rehabilitation counselors, adult service providers, 
parents, and paraprof essionals to determine an individual's 
general skill level in relation to standards suggested as 
important for success in such occupations as the food service 
industry, janitorial work, and light indu5?trial occupations. 
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Figure 1 
The Job Inventory Process 



Identify the general work and 
social skill requirements of a 
community job. 

Observe worker performance to 
analyze the component tasks of 
the job and establish normal 
levels of productivity • 



Determine the need for additional 
community support services, for 
example, transportation and 
independent living. 

— i 

Develop an individualized training 
program and plan continuous 
evaluation procedures. 
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The primary purpose of the instrument is to assist in the job 
inventory process by providing a starting point for the 
development of a supported employment program in food service, 
janitorial, or light industrial occupations • The VACG also 
provides functional training objectives for school-aged students 
with handicaps as they move toward competitive employment 
opportunities. The VACG has proven to be a useful evaluation 
instrument in ongoing supported employment programs. 

Vogelsberg (1986) has described the role of the VACG in 
supported employment evaluation in Vermont. (The VACG is one of 
two instruments utilized in the evaluation phase of the Vermont 
program; the other is a locally developed tool, the Individual 
Skill Inventory . ) The VACG is administered to provide 
curriculum recommendations for individuals who cannot be 
immediately employed. The recommendations, in the form of 
specific training objectives, are forwarded to adult service and 
local education agencies for incorporation into an individual's 
program plan. The Individual Skill Inventory provides the 
structure for completing the job inventory process for specific 
positions in the community. Vogelsberg (1986) has stressed that 
factors such as transportation, parental support, and agency 
cooperation are also evaluated and play an important role in 
identifying the best candidates for supported employment. 

Menchetti and Rusch (1987) have investigated selected 
psychometric properties of the VACG. These included reliability 
issues of score stability, internal consistency, and interrater 
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agreement. The capability of VACG domain scores to discriminate 
between groups of handicapped and nonhandicapped workers was 
also examined as a validity concern. The reliability data 
obtained for the VACG also compared favorably with similar 
information reported for other vocational rating scales. The 
VACG validation data indicated that domain scores differentiate 
between groups of handicapped and nonhandicapped workers 
employed in service occupations. 

Studies investigating the psychometric properties of 
instruments with an ecological analysis orientation are 
important for a number of reasons. Research of this kind 
signals a return to the rigorous scientific regimen of test 
development and validation suggested by pioneers in the field 
such as R. L. Thorndike. Renewed emphasis on empiricism will 
enhance the field of vocational evaluation and improve the 
adequacy of measurement techniques. 

The job inventory process and instruments such as the VACG, 
which are based on an empirical analysis of the employment 
ecology, provide a strategy for the d ^lopment of useful 
evaluation procedures. Professionals interested in providing 
effective supported employment services must use this strategy 
in their evaluation programs. 

Vocational Evaluation for Supported Employment 

Meeting the challenge of supported employment will require 
an evaluation program that is designed to meet the needs of 
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local labor markets and the individual seeking a job. Several 
steps must be followed by evaluators to assure that the data 
they obtain are functionally related to community work 
opportunities and the employment needs of individuals with 
handicaps. These include evaluating local labor markets, 
evaluating potential employees' vocational and related skill 
needs, developing individualized training plans, continously 
monitoring program efforts, and evaluating employer satisfaction 
on a regular basis. 



Evaluating Local Labor Markets 

An initial step \n the development of supported employment 
evaluation programs should be the identification of the needs of 
community employers. Evaluators can utilize techniques such as 
employer surveys and job analysis (Martin, 1986) to assess 
target work environments (e.g., restaurants, factories, 
offices.) The information obtained can be used to specify both 
general and specific skill requirements of locally available 
jobs. If this step is not practical at the local level, 
instruments such as the VACG will provide useful information 
about the skill requirements of service and light industrial 
occupations. 



Evaluating Individual Needs 

After obtaining information about marketable community work 
skills, the evaluator must turn his or her attention to the 
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individual. Evaluators should determine a potential employee's 
current skill level in relation to the community- referenced 
standards identified with local surveys or with the VACG. 
Locally developed skill checklists or the VACG Inventory can be 
used for this purpose. In addition, employment-related factors 
such as transportation, medical needs, or economic 
considerations must be assessed. Usually, prior assessment 
suiTuuaries or school records will contain this important 
information. 

Plan Indi ^ " - lal Trainin cr 

Training ob.iectives mv be specified for each individual 
seeking supported employment. Many school programs have begun 
writing individualized transition plans (iTPs) for high school 
students. Voc'tional rehabilitation professionals utilize an 
individualized written rehabilitation plan or IWRP to c-pecify 
services. The ITP and IWRP provide vehicles for specifying 
training objectives related to supported employment. Vocational 
evaluators can write training objectives for each of the skills 
included on a locally developed skilx checklist or use the VACG 
Curriculum Guide to target objectives for inclusion on an 
individual's program plan. 



C ontinously Monitor Progress 

In supported employment programs, the evaluator's role does 
not end with the development of a program plan, planning and 
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training efforts must be closely connected and mutually 
beneficial. Individualized program plans (i.e., ITPs, IWRPs) 
should include recommendations for continous monitoring of 
worker performance. Behavioral observation techniques and 
repeated measures experimental designs such as the multiple 
baseline and changing criterion are useful tools for the 
analysis of work behavior (Agran^ 1986). Vocational evaluators 
must be f amili ar *with these techniques if their efforts are co 
facilitate the provision of effective services. A discussion of 
behavioral assessment methodology is beyond the scope of this 
chapter, but the reader may consult one of the excellent sources 
of information on this topic (Agran, 1986; Bates & Hanson, 1983; 
Kazdin, 1982, Kratochwill, 1978; Rusch & Mithaug, 1980; Wehman, 
1981). Continous assessment of worker performance is needed to 
facilitate supported employment decisions such as the need to 
provide retraining and the appropriate time to withdraw 
intervention procedures. This kind of evaluation is crucial to 
the success of 'supported employment programs. 

Evaluating Employer ^Satisfaction 

^'S 

The final step in evaluation should be an assessment of the 
employer's satisfaction with supported employment. Employers 
are important participants in supported employment, and the most 
successful programs have evaluated their perception of training 
goals, procedures, and outcomes. Information regarding employer 
satisfaction enables programs to be responsive to the needs of 
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the local labor market and increases the likelihood that these 
markets will be accessible to persons with handicaps. 



Recommended Changes in Current Evaluation 
and Eligibility Procedures 



We have attempted to build a case for change in current 
rehabilitation and education evaluation procedures. In 
addition, recommendations and examples of a more functional 
approach to vocational evaluation of persons with handicaps have 
been presented. There is no doubt that, once put into 
widespread practice, these functional evaluation techniques will 
also replace traditional ideai about eligibility for employment 
services. In summation, we suggest the following changes in the 
current procedures used to determine whether or not an 
individual has potentional for gainful employment. 
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1. Vocational evaluation techniques must move away 
fro m methods that measure general aptitudes and 
work habits that have never been shown to be 
related to the successful vocational adjustment 
of individuals with handicaps. 

2. Service providers must abandon eligibility 
procedures based upon invalid predictions of 
general employment potential. 
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Vocational evaluation techniques must be based 
upcn an empirical analysis of the local labor 
market. Techniques and instruments such as a job 
inventory and the VACG may prove useful. 

Vocational evaluation efforts must repult in 
information that facilitates program planning, is 
community referenced, and assists In instructional 
decision making. Evaluation in supported 
employment programs must be continous to assure 
that retraining can be provided when needed and 
that employer satisfaction can be measured. 

Service providers must base eligibility aetermination 
for supported employment programs on 30b inventory 
data, job availability, presence of supportive family 
members, transporation factors, and the econ^ Ic 
consequences of community employment on tlie target 
individual. 

Vocational evaluation professionals must rediscover 
the rigorous empirical methods suggested by the 
pioneers to develop new techniques for use with 
handicapped persons. 
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The Use of Srandardized Assessment 
in Supported Employment 
Lizanne DeStefano 



The widespread implementation of supported employment programs 
across the country has led to concern in the field about the most 
useful and appropricxte assessment strategy to be used in supported 
employment settings. Traditionally, vocational training programs 
have relied upon standardized techniques that combine aptitude 
tests, interest inventories, and norm- referenced instrUi-nents to 
generate individual profiles of vocational performance and potential 
(Gay lord-Ross, 1985). Although such instruments have proven useful, 
there is some question about their appropriateness for individuals 
with severe handicaps, the primary target group ^or supported 
employment. This concern has led to the current move toward a 
contemporary assessrr-^nt approach that relies heavily upon 
situational assessment, ecological inventories, and 
criterion-referenced measures (Halpern, Lehman, Irvin, & Heiry, 
1982; Menchetti, Rusch, & Owens, 1983). 

This author takes the position that despite the presence of 
unique population characteristics, both traditional and contemporary 
assessment techniques contribute valuable information for decision 
making and documentation within the 



To appear in F. R. Rusch (Ed.) (forthcoming). Supported competitive 
employment issues and strategies (2nd Ed.). Baltimore: Pau] H. 
Brookes. 
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supported employment model. The task that service providers 
face is to devise an assessment package that adopts the best of 
both traditional and contemporary approaches. The package 
should be time and cost efficient and should provide information 
that is maximally useful for (a) identification of the target 
population, (b) program planning and placement, (c) monitoring 
worker progress, and (d) program evaluation (Lynn & DeStefano, 
1986) . 

This article presents a number of standardized techniques, 
discusses their strengths and weaknesses for use with 
individuals with severe handicaps, and gives some guidelines for 
the selection and use of standardized tests in supported 
employment. For our purposes we shall define a standardized 
test as either a norm- or criterion-referenced instrument for 
which reliability, validity, and administration procedures have 
been well documented. Although this paper focuses on the use of 
standardized tests, it should be made clear that these tests 
constitute only one part of the assessment data to be 
collected. No diagnostic planning or placement decision should 
be made on the basis of standardized test data alone. 
Situational assessment holds an equally important role in 
completing the assessment picture. 



Why Use Standardized Tests? 



Situational assessment, observation of performance on the 
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job and employer/co-worker evaluation provide a great deal of 
the situation-specific information that is necessary for making 
decisions and evaluating the success of a supported employment 
placement. Standardized assessment instruments complement these 
contemporary methods by providing generalized information that 
can be used across setting, across individuals, and across 
time. The comparative capability of standardized test 
information makes it useful in obtaining initial information 
about a worker, in describing the population served by supported 
employment, and in evaluating the impact of supported employment 
on an individual's adaptive functioning and overall quality of 
life. These uses will be discussed below. 

Regardless of their orientation, all approaches to training 
share the need for some preliminary information about the 
individual to be trained. The use of standardized instruments 
offers ti^e advantage of providing descriptive information about 
an individual across a number of competency areas. The 
information can be obtained in a few hours, and performance is 
summarized in standardized scores, percentile ranks, and age 
equivalents, which are easily communicated to the educational, 
vocational, and rehabilitation personnel involved in the process 
of planning for training. These standardized scores are 
obtained by comparing the examinee's performance with the 
average performance of persons in the standardization sample. 
In many instances it may be of interest to com' e^e the results 
with those of a cohort made up of nonhandicapped individuals. 
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In other instances it may be uore valuable to compare the 
examinee's performance with that of individuals with similar 
handicaps. For example, it may be helpful to compare an 
individual's scores on a vocational skills inventory with local 
production norms for nonhandicapped co-workers in addition to 
production rates of enclave members with handicaps. Using both 
of these comparisons, a job coach can anticipate the level of 
assistance necessary to attain maximum productivity. Many 
standardized tests offer boui these options as aids to 
interpreting individual performances. Administration of a short 
battery of standardized tests upon entry to a program can 
quicKly identify skil" areas where additional inquiry is needed 
in the form of observ^cion, situational assessment, ecological 
inventories, or interview. 

A comprehensive description of the population served is an 
essential elerr nt of program evaluation, replication, and 
dissemiuation. Standardized test data can be used to summarize 
the levels of functioning and skill levels of workers in a 
supported en.ployment program, so that persons unfamiliar with 
the program can quickly get a sense of the severity of the 
disability served and of the relevance of the program to their 
own client group. 

Standardized tests also offer the advantage of reliability 
and validity documentation, which not only p-c ides a means of 
interpreting the results of th-se tests, but show their 
limitations and the degree to which they are consistent acres?: 
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time and across testers. Administration procedures and 
materials are outlined clearly so that the sts are given the 
same way each time. With these assurances it is possible to 
compare data from one administration with data from another. In 
this way, standardized tests can be used in a repeated-measures 
fashion to assess change or stability in a particular 
characteristic over time or as a result of an intervention. 
Obviously, the most logical way to assess the effectiveness of a 
supported employment program is to look at job ter^ure, wages and 
benefits earned, and employer satisfaction. It is desirable, 
however, to assess the broader impact of participation in 
supported employment by examining chnages in adaptive 
functioning, socialization, and lifestyle satisfaction. 
Standardized tests serve as one way to measure change in these 
contructs. Change can be measured on an individual basis to 
provMe information on how effectively an individualized 
rehabilitation or vocational service plan is meeting its goals 
and to suggest areas for future planning. Longitudinal data can 
also be collected and summarized for all workers in a ^uj^ported 
employment program as a measure of overall program 
effectiveness. Data from similar standardized test batteries 
can be v '•r^d in a comparative analysis of many supported 
employment programs or can be aggregated a:.d used to represent 
the characteristics of th^ population served collectively by 
supported employment programs in a state or region of the 
country. 
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Although standardized tests can contribute significantly to 
understanding and monitoring an Individual worker or an entire 
supported employment program, there are many considerations in 
selecting tests that are both appropriate and maxiinally useful. 
In the next section I discuss the usefulness of several 
commercially available standardized tests in a supported 
employment program. 



Guidelines for the Selection 
Standardize! Tests 



There are several guidelines that should be followed when 
selecting one standardized test over another. The following 
guidelines are intended to help structure selection. One of the 
first considerations relates to why a test is used. The content 
of the test should be clearly related to the p urpose of the 
test. For example, if a test of adaptive behavior is needed to 
obtain information upon level of community integration it 
should be determined that a substantial number of items on the 
test are related to community integration and that those items 
are of the type that are relevant for the individual. It is not 
enough to accept the test label or publisher's descriptions as 
accurate accounts of test content. Personal inspection is the 
surest way to determine if test content meets assessment needs. 

Although the 'characteristics of the standardization sample 
are important to consider in all test selection, nowhere is this 
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information more important than in the use of tests with special 
populations. First, the extent to which members of the special 
population were included in the standardardization sample should 
be determined, and information should be available regarding 
their performance in terms cf norms and standard error of 
measurement. If no members of the special population wore 
included in the standarization sample, it may be difficult to 
interpret assessment findings accurately. 

Information about the reliability and validity of the test 
should be stated clearly in the examiner's marual or in 
published research. Inter-rater, test-retest, alternate forms, 
and internal consistency reliability data should be provided for 
the entire norming group as well as for any special 
populations. Information about validity should include 
information from studies that include correlational analyses 
with other s'.tandardized measures, factor analytic studies, 
discriminative analyses, or studies of the ability of the test 
to predict fitatus on some outcome variable. Again, validity 
information for the special population that is relevant to the 
assessmenc putir->ses is highly desirable. Documentativ^n of the 
validity of a test is important because it provides evidence 
that a test actually measures what it purports to measure or 
that it is useful for its intended purpose. 

The time necessary to administer the test should be 
considered in terms of the importance of the information 
gained. Since the primary goal of any supported employment 
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program is to obtain and maintain employment, any assessment 
program that interferes with that process dnd does not 
contribute information sufficient to warrant that disruption ±3 
inappropriate • For example, if workers who are placed must miss 
work to complete follow-up assessment for evaluation purposes, 
something is in error. The fact that worker is working 
provides greater evidence regarding the success of the program 
than gain scores on a standardized measure. 

Any assessment done in the employment environment should not 
disrupt that environment. The information gained should be 
obtained through routine disruptions (e.g., during breaks, 
before work, after work), or during scheduled brief periods. 

Each standardized instrument has specific requirements for 
the training of individuals who will be administering it. 
Personnel with the appropriate qualifications should be 
available for sufficient time periods to administer, interpret, 
and report test findings. 



Standardized Tests and the 
Constructs They Measure 



Because the successful placement of an individual into an 
integrated work setting depends upon a multitude of interrelated 
personal and situational variables, the range of personal skills 
and attributes to be assessed with standardized measures is 
equally broad. Linn and DeStefano (1986) reviewed assessment 
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practices of more than 100 federally funded transition programs 
and identified 12 areas of assessment common to the majority of 
these projects: general ability/intelligence, adaptive behavior, 
vocational skill, career interests and awareness, language, 
social skills, academic skills, special abilities, survival 
skills, daily living skills, motor skills/dexterity, and 
lifestyle/consumer satisfaction. In a subsequent study 
(DeStefano & Linn, 1987) these 12 areas were consolidated into 
eight broad categories: general ability/ intelligence ; adaptive 
behavior; vocational skills; career interest and awareness; 
language; academic achievement; special abilities; and lifestyle 
satisf actiori, (Because of its multifaceted definition the 
construct of adaptive behavior has been used to subsume the 
categories of social skills, daily living skills, and survival 
skills in the new categorization scherre.) 

These eight areas will be used as the framework for a 
discussion of commercially available standardized tests. An 
ideal test battery should include measures in any area that is 
relevant to the functioning level and employment options of the 
individual. Limited resources may necessitate restricting 
assessment to those areas of highest priority dictated by 
program characteristics or client needs. Program personnel 
should review instruments from several areas in combination with 
each other to design a test battery or a set of batteries that 
best meet the identification, placement, planning, monitoring^ 
and evaluation needs of the program. The following overviews 
each of the eight assessment areas. 

o 74 
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General Ability/Intelligence 

Intelligence testing began in the United States in the early 
1900s and has played an important role in education and 
psychology from that moment forward. At some point in their 
lives most individuals with handicaps have taken an intelligence 
test. Most often trie test has been used diagnostically to 
assign to the individual some educational or psychiatric 
description such as learning disabilities, mental retardation, 
or neurological impairment, or to substantiate the original 
diagnosis during periodic reviews. Out of administrative 
necessity, intelligence scores have come to be used as criteria 
for the provision or denial of services. It is often this 
gatekeeping function of intelligence testing that is emphasized 
at the expense of its equally important identification and 
informational capabilities. These latter functions hold the 
most relevance for supported employment programs. 

Supported employment is a new day option that is targeted 
for persons with severe handicaps. In many instances the 
handicap will be characterized by a deficit in intellectual 
functioning, as in the case of persons with mental retardation, 
the second largest group seen for vocational rehabilitation 
services (English, Oberle, & Byrne, 1979). For persons with 
mental retardation, intelligence tests communicate the *'lev3l" 
of handicap quickly and serve as the basis for most diagnosis in 
this area. 

The definition of mental retardation that is proposed by the 
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American Association on Mental Deficiency {\AMD) and accepted by 
Diagnostic and Statistical Manual of Mental Disorders (1980) 
(DSM-III) is the one tnat is most frequently U'-ed: 

Mental retardation refers to significantly 
subaverage general intellectual functioning 
existing concurrently with deficits in 
adaptive behavior. (Grossman, p. 11) 

The term s ignificantly subaverage denotes that the score must be 
at least 2 standcird deviations below the mean IQ of the 
population as a wbole. By convention th^ mean IQ is 100, and 
although the standard deviation might vary slightly from test to 
test, it is typically 15 points, so the IQ cutoff point for a 
diagnosis of mental retardation is generally 70 (two standard 
deviations below the population norm of 100). However, because 
no test is absolutely reliable, DSM-III and AAMD express the 
upper limit of mental retardation as a range from 65 to 75, 
depending upon clinical juagment to determine the degree of 
impairment. Within the range of mental retardation there are 
several subcategories: 

Mild - IQ range 55 to 69; 

Moderate - IQ range 40 to 54; 
Severe - IQ range 25 to 39; and 

Profound - IQ runge less than 25. 
Supported employment prograir^s should have on record an IQ 
score for each client in their program. The IQ score should be 
no more than 3 years old, because numerous studies indicate 
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ability scores may, and probably will, change over the years 
(Carvajal, Lane, & Gay, 1984; Givens & Davis, 1984; Zimmerman, 
Covin, Woo-Sam, Lotz, & Bley, 1984). IQs serve as a means to 
summarize information regarding persons participating in a 
program as an aid to administrators and planners of service. In 
order to identify the population served by a supported 
work/ employment program, the program should be interested in ^he 
range of IQs of the population sei ^ed, the mean and modal IQ 
score of that population, and any changes in mean IQ score of 
the population over time. For example, in Illinois, the 
population served by supported employment had an IQ range of 18 
to 74, with a mean of 53. Lagomarcino (1986) reported serving 
persons with IQs of less than 20 using a supported employment 
model. In a recent study of the Illinois Supported Employment 
Project (Schutz, Trach, & Winking, 1987) reported that the m-^an 
IQ of persons served by supported employment projects in 
111^ lois decreased steadily over an 18-month period. This 
decrease was explained by the fact that as local providers 
learned how to implement the supported work model, they began to 
serve lower-functioning persons. This finding may be common in 
newly developing supported employment programs. 

Many validity studies have been done to determine the 
relationship between IQ and academic achievement, but little is 
known of how well IQ can predict success on the job. In the 
area of supported employment, studies have begun to investigate 
the relationship between IQ and various components of the 
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supported employment model (Tracn, Rusch, & DeStefano, 1987). 
Trach et al. (x987) found that individuals of lower intellectual 
functioning received more job-matching and maintainence services 
than persons with higher intelligence scores. These results 
suggest that alternative service plans can be expected based 
upon intelligence scores. This type of prediction based upon 
anticipation of individual needs and directed at improving 
service delivery is in stark contrast to the gatekeeping 
functions typically associated with intelligence tests 
(Menchetti & Rusch, 1987). 

The following brief discussion covers several of the most 
frequently administered measures of general ability/intelligence 
used to 6issess persons with severe disabilities (Linn & 
DeStefano, 1986; Maloney & Ward, 1979; Roszkowski & Bean, 1980) 
and describes the advantages and limitations of their use in 
supported employment programs. 

The Wechsler Adult Intelligence Scale - Revised (WAIS-R) 
(Wechsler, 1980). Although standardized on a normal population 
(N = 1,880) ages 16-74, the WAIS-R has been used extensively 
with persons with all types of disabilities (Zimmerman et al. , 
1984; Zimmerman & Woo Sam, 1972). In fact, the WAIS-R has 
replaced the WAIS as 'cho most commonly used test for adults with 
mild to moderate mental retarda+ on. The reliability anc" 
validity of the WAIS-R are well established (Wechsler, 1981), 
and administratioa procedures are clear. Extensive training, 
supervision, and certification are necessary to become an 
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examiner. In addition to obtaining an IQ score, the examiner is 
able to differentiate between an individual's verbal and 
performance abilities. 

It is important to note that the WAIS^R was constructed to 
assess IQs of 50 or higher. When the test is administered to 
persons with IQs below 50, they typically are unable to succeed 
even on the easiest items, a -^loor effect that eliminates the 
possibility of obtaining a meaningful IQ or an interpretable 
profile. Consequently, the WAIS-R is not an appropriate measure 
of intelligence for individuals in the low moderate to profound 
range of mental retardation. The examiner should also be aware 
that the WAIS-R requires both verbal and motor responses, and 
several subtests require fine auditory and visual 
discrimination. For these reasons the WAIS-R may not be 
appropriate for some individuals with orthopedic, speech, or 
sensory impairments. 

The Stanf ord-Binet Intelligence Scale: Form L-M (Terman & 
Merrill, 1973). The Stanf ord-Binet is still the instrument of 
choice in evaluating the intellectual functioning of adults in 
the moderate to profound range of mental retardation. Because 
the Stanf ord-Binet spans age ranges from 2 to adult, it is 
possible to obtain a meaningful IQ as low as 36 and mental age 
estimates for persons who function below this IQ- The 
reliability and valiaity of the Binet have been well 
doc'omented. Administration procedures are clear. Extensive 
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training, supervision, and certification are necessary to become 
an examiner. Even though it is the best test available for 
assessing low-functioning individuals, the Binet has a number of 
limitations when it is used with clients with mental 
retardation. Most important, the test is outdated. The most 
recent norms (Thorndike, 1972) are 15 years old. Second, the 
test produces only a single summary score, which reflects a 
mixture of verbal and performance items. There is no mechanism 
for profile analysis. Last, the test has been criticized as 
heavily loaded with verbal tasks (Sattler, 1982) and therefore 
can be misleading for use with persons of low language ability. 
A recent revision, the Stanford-Binet , Fourth Edition 
(Thorndike, Hagan, & Sattler, 1985), which addresses each of 
these weaknesses, is currently available. Unfortunately, adult 
norms for the test have not yet been released, and its validity 
for use v/ith special populations has not been established. 
Consequently, despite considerable limitations, the 
Stanford-Binet: Form L-M remains the best option for assessing 
the general ability of lower-functioning persons. 
The Slosson Intelligence Test (SIT) (Slosson, 1961). The SIT i- 
a short screening test that consists of content drawn from the 
Stanford-Binet, and it also heavily emphasizes language. The 
SIT vtais standardized on children ages 4-18. The standardization 
sample was limited and is out of date. The reliability and 
validity of the mental ages and IQ scores obtained by the test 
have not been reportea. The SIT may be useful as a rapidly 
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administered screening test, but the absence of adult norms, the 
lack of reliability and validity data, and the single score 
format limit its value. The SIT is not a substitute for a 
VhIS-R or a Binet and should not be considered for use in any 
testing battery associated with supported employment. 

Summary . Intelligence test scores have proven useful for 
describing an individual or a group in terms of general level of 
functioning in order to identify a targat population or to 
evaluate or monitor a program (Trach et al., 1987). 
Intelligence scores should not serve as a basis for exclusion 
from services, but may indicate the nature and extent of 
appropriate services in a supported employment model (Trach et 
al., 1987). Because of their composite nature and resistance to 
change, intelligence scores have less utility when considering 
day-to-day program planning needs or moni coring an individual's 
progress over time. Another group of standardized tests have 
direct applications for program planning and longitudinal 
follow-up. These are the adaptive behavior measures. 

Adaptive Behavior 

The construct of adaptive behavior is defined as the 
performance of the daily acti^'ities required for personal and 
social sufficiency (Sparrow, Balla, & Cicchetti, 1984). 

The AAMD manual lists two major facets of adaptive behavior: 
1. The degree to which the individual is able to function 
and maintain himself /herself independently, and 
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2. The degree to which he or she meets sacisf actorily the 
culturally impose'l demands of personal and social 
responsibility (Grossman, 1977). 
These dual facets reflect clearly the values of independence and 
integration which are associated with supported employment and 
adult services. 

Like deficits in intelligence, deficits in adaptive behavior 
must be present for a diagnosis of mental retardation to be 
made. Thus, adaptive behavior scores are useful in identifying 
members of this target population for a supported employment 
program. Adaptive behavior deficits are also seen in persons 
with mental illness, physical handicaps, and sensory 
impairments. As supported er.ployi'nent programs expand to include 
these populations, the construct of adaptive behavior may 
provide a more m.e^ningful description of the level of 
functioning of the popula'-ion served than intelligence test 
scores . 

Unlike the construct of intelligence, which provides summary 
information that is useful to communicate general population 
abilities but has little relevance for program planning and 
monitoring worker progress, adaptive behavior comprises several 
domains that have direct relevance for training. Although 
content varies across tests, most tests of adaptive behavior 
contain items that measure m otor skills , including fine and 
gross motor skills; communication skills , including receptive 
language, expressive language, and written and oral 
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communication; social skills , including peer relationships, 
leisure activities, and coping skills; personal living skills , 
including eatxng and meal preparation, toileting, dressing, 
personal selt-care, and housekeeping; and community living 
skills , toileting, dressing, personal self -care, and 
Housekeeping; and da i ly living skills , including time and 
punctuality, money and ranking, vocational skills, and 
mobility. Many of these sk:.ll areas are impoitan^. for 
employment success and overall community integration. 

T'r^'^ts of- adaptive behav. r can serve man^ purposes in 
supported employment. Individua] ' ^ed training plans in the 
areas of personal hygiene, time management, or finance can be 
developed which are based upon an item analysis of an 
individual's performance on an adaptive behavior measure 
overlaid with an analysis of the requisites ^f the job. 
Readministrations of the same test annually or biannually can be 
used to measure an individual o attainment of goals stated in 
the plan. Changes in adaptive behavior may occur for reasons 
other than planned intervention, however. As individuals become 
successful] ' integraced into the community in leisure, work, and 
residentic\ settings, increases in adaptive behavior scores will 
occur. For this reason changes in raw scores can be used to 
document the adaptive progress of an individual in supported 
employment or the overall influence of integration into programs 
other t' an employment, such as leisure and recreational 
activities. This measure can be used for program evaluation 
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purposes to document increases in level of independent living as 
a result of participation in supported employiiient . 

Adaptive behavior instruments* have proliferated in the last 
few years. In 1977, Walls and Werner lisred 136 commercially 
available measures of adaptive behavior. Increasingly, new tests 
or measures inee*: standards of psychometric quality ai:d combine 
both criterion and normative referencing. Most K^easi.res involve 
the cooperation of a knowledgeable informant, typicall/ a parent 
or teacher. .dditional information Uiay be needed from direct 
observation. The examinee alone is less frequently encouraged 
to serve as his own informant. Several of the most widely used 
and promising measures are discussed below. 
The aAMD Adaptive Behavior Scale (ABS) (Nihira, Foster, 
Shellhas, & Leland, 1975). The ABS was developed at the request 
of AAMD in 1969 and revised in 1974 for use with 
institutionalized children c.nd adults. Part One consists of 66 
items across 10 separate domains, including vocational 
activity. The domains are scored separately, resulting in a 
profile of adaptive skills presented as percentiles. Although 
"vocational activity" is considered separately, the domain 
consisf^' of only three items: job complexity, job performance, 
and work habits. Rating of job complexity is limited to a) 
performs no work at al^ b) performs simple work; or c) performs 
a job requiring the use of tools or machinery. Part Two of tne 
ABS consists of 44 ite as across 14 domains designed to measure 
maladaptive behavior. The maladaptive scalei are useful to 
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identify behavior and emotional problems that are independent of 
the intellectual level of functioning. 

The ABS was standardized on 4,000 institutionalized 
individuals with mental retardation, ages 3-69. Percentile 
norms are available on 11 age groups. The ABS is at present the 
most widely used of the adaptive behavior measures, yet its 
limitations are considerable. In contrast to the simplicity of 
a summary score, the ABS produces a profile that is complex o.xid 
difficult to interpret. The use of norm, based solexy on 
institutionalized subjects is inappropriate for making j^i^graents 
about noninstitu' ionalized individuals. Given the number of 
well -standardized tests of adaptive behavior that have recently 
appeared on the marke^., the ABS is no longer the instrument of 
choice for assessing adaptive behavior in integrated settings. 
The Vine land Adaptive Behavior Scale (VABS) (Sparrow, Balla, & 
Cicchetti, 1984). This revision of the original Vineland Social 
Maturity Scale (Doll, 1953) attempted to answer several 
objections associated with the original version as well as w: th 
the ABS described above. The VABS consists of a Surv Form of 
297 items and au Expanded Form of 572 items. The instrument 
covers age levels from birth through 18 and low-functioning 
adults. Items are grouped into four domains: communicaf^ on, 
daily living skills, socialization, and motor skills. A m>ii\ber 
of items across the various domains directly measure vocational 
aajurtment. The VABS ^Iso has a second section of 36 i^ems that 
alow for assessment of problem behaviors. 



standardized Assessment 

75 

The standardization of the VABS included 3,000 normal 
children from birth to li; with supplemental standardization 
samples of children with various handicaps (N = 680) and adults 
(N = 2,202). Norms for normal adults are not available. 
Adults scores are derived with the use of norms for 
18-year-olds. Each of the four domains and an Adaptive Behavior 
Composite score can be expressed in standard scores, percentile 
ranks, ^nd age equivalents. 

Substantial training is necessary to administer the Vineland 
using a semi-structured interview technique. The interview 
requires 40 to 90 minutes administration time. 

The lack of adult norms detracts somewhat from the 
usefulness of the VABS in supported employment settings. 
Pe/sonnel qualifications and lengthy administration time may 
also detract from its usefulness to some proirams. However, if 
comparison with the normal population is not critical and 
resources are adequate, the substantial number of items that 
pertain to vocational adjustment and its excellent 
standardization warrant consideration cf the VaBS. 
Scales of Independent Behavior (SIB) (Bruininks, Woodcock, 
Weatherman, & Hill, 1984). The SIB offers an approach to 
assessing adaptive behavior that provides information that may 
be suited for supported employment. The SIB utilizes 266 Icems 
to assess 4 cxus^ers of behavior: motor skills, social 
interaction and comiTiunication skills, personal living skills, 
and community living skills. The last category consists o^ 64 
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items related to successful employment. Scores can be expressed 
as standard scores, percentile ranks, and age scores, based on a 
standardization sample of 1,764 normal individuals ranging from 
infants to those more than 40 years of age. The SIB also 
contains a detailed scale that measures internalized, 
externalized, and asocial maladaptive behavior and assesses both 
frequency of occurrence and severity of the problem. The SIB 
appears to be one of the only adaptive behavior scales with 
normative data for normal adults, which enhances its value for 
assessing adjustment in the context of employment • 
Inventory for Client and Agency Planning (ICAP) (Bruininks, 
Hill, Weatherman, & Woodcock, 1936). The ICAP offers a quick, 
convenient way to assess client characteristics and to determine 
service needs. An abbreviated form of the SIB, the ICAP is 
appropriate for infancy through adulthood; it takes only 20 to 
25 mjnutes to administer. The ICAP shares common norms with the 
SIB which was standardized on 1,764 subjects in 40 communities 
across the United States. Covering the domains of motor skills, 
social and communication skills, and personal living and 
community living skills, the ICAP yields the following scores: 
age equivalents, standard scores, instructional training range, 
relative performance index, maladaptive behavior indexes, 
percentile ranks, normal curve equivalents, service levels, and 
service scores. The ICAP can be administered by a wide variety 
of personnel and can be used to measare an individual's present 
status, to identify client needs, to set service goals, to 
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monitor progress, and to evaluate program effectiveness. 
Street Survival Skills Qaestlonnaire (SSS Q) (Likenborn & 
McCarron, 1980). The SSSQ is designed as a measure of communitv 
living and prevocational skills for adolescents and adults with 
mental retardation. Item content includes basic concepts, 
functional signs, tools, domestic management, 
health/safety/first aid, public service, time, money, and 
measurement. The SSSQ is administered directly to the 
individual, using a multiple choice format in which the examinee 
points to the correct response. The test was normed on 
adolescents and adults with mental retardation (N = 500, ages 
15-55) and normal adolescents (N = 200, ages 14-18). 
Te:^"-retest reliability is high at .94. Validity data are 
limited. Although its small standardization sample size and 
limited content make it an unaccepted measure of adaptive 
behavior for purposes of diagnosis, the SSGQ may provide 
information that is useful for prograin planning and theiefore 
may be considered for use in supported e.Tiployrr.ent . 

Summary . Tests of adaptive behavior provide information 
about an individual's motor skills, daily living skills, social 
skills, and communication skills. Many adaptive behavior tests 
have items relating directly to wor- performance. This 
information can be used in conjunction with job analysis datdi 
and other situational assessment information o choose 
placements and develop training plans. Composite scores on 
tests of adaptive behavior can be used as indicators of an 
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individual's level of independence and integration. Cnanges i^ 
raw scores over time can be used to document growth in 
independent living. 3oth of unese uses of adaptive behavior 
data have direct implications for evaluating supported 
employment programs. 

Although tests of adaptive behavior include items related to 
work, another group of tests — tests of vocational skills 
focus solely on this area. 

Vocational Skills 

The use of work samples as vocational assessment tools 
"represents the most popular assessment strategy in use tc^.ay" 
(Menchetti, Rusch, & Owens, 1933, p. 83). There have been many 
standardized vocational assessment instruments that have 
attempted to generate individual profiles of vocational 
performance and potential. Although such instruments have 
Droven useful in some circumstances, they are of questionable 
utility for most of the individuals served by supported 
employment programs. Waxk {1985} reviewed the performance of 
persons with mental retardation, neurological impairments, 
orthopedic impairments, and visual impairments on 14 work- sample 
systems. He concluded that not one of the 14 corpmercjal systems 
was without fault nor could be said to function as a complete 
assessment tool. The limitations aro numerous. Commercial work 
samples are noz readily adaptable to a supported emploi'r3nt 
setting. The material required is expensive and ciimbersome. 

8:i 



standardized Assessment 

79 

The administration time ^-nay cover days or weeks. 

Work samples have limited predictive value for a por>ulaticn 
of persons with retardation. These systems have been faulted 
because they measure products of prior learnincr and neglect 
progress made on the job itself. They often fail to duplicate 
the exact characteristics of specific job clusters and bypass 
social aspects tnat are most often associated with job retention 
(Greenspan & Schoultz, 1981; Schalock & Harper, 1978). They 
provide liti.le substantive information to be used in developing 
individualized training plans. In fact, the validity of work 
sample assessment has not been established even for individuals 
with mild mental retardation (Irvin, Gersten, Taylor, Close, & 
Bellamy, 1981). 

Because of the severe limitations of the work sampl3 system, 
other types of vocational assessment instruments are beginning 
to LB developed. One such instrument, the skill inventory, is a 
low inference approach to vocational assessment because it 
attempts to attain specific behavioral descriptions of the 
individual in a particular work setting rather tha . inferring 
ability from sampling a limited range skills in a contrived 
setting. To use such an inventory effectively, skill 
requirements for specif 'j jobs in the community should be known 
in advance cind us<^d as the criteria to analyze inventory data 
and to determine the degree of fit b'^tween worker and job. 

Social validation methodology plays an integral role in the 
development of these kindc. of assessment instruments. This 
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methodology involves asking a numbe^ of "experts" to look at the 
instrument to determine if the items and the criteria used to 
evaluate them are relevant, representative, properly stated, and 
correctly placed. This "expert" review can include the opinions 
of supervisors nnd persons who have actually performed the tasks 
being evaluated in a competitive employment situation. 
Vocational Assessment and Cuiriculum Guide (VACG) (Rucch, 
Schutz, Mithaug, Stewart, & Mar, 1982), The VACG is a socially 
and psychorretrically validated instrument that assess^^s broad 
classes of vocational and social skills of individuals preparing 
for competitive employment. The skills assessed are those that 
employers considered important for entry- level employment in 
food, janitorial, and maid services, and light industrial 
occupations. The VACG can be used to assess skill deficits and 
strengths in terms of competitive employment expectations and to 
pre :.ribe training goals decigned to reduce identified 
deficits. Repeated ac'ministrations of the VACG can be used to 
monitor individual progress and to evaluate program 
effectiveness by assessing progress toward identified goals. 
The VACG contains 42 items across 10 subcategories: 
attendance/eiiduraiice , independence , production , l»3arning , 
behavior, communication skills, social skills, self-help skills, 
grooming/eating, and math Each item requires that one or more 
questions be answered regcrding how the worker 

::haracteristically responds in a given work situation. The VACG 
consists of a manual, an inventory, and a curriculum guide. 
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Reliability and validity studies have recently been 
completed using the VACG. The reliability data obtained by 
Menchetti and Rusch (1987) indicated that the VACG provided 
stable, consistent, and accurate measurement of skills. The 
VACG validation data indicated that domain scores differentiate 
betv;ee ^ groups of workers with handicaps employed in sheltered 
settings and groups of handicapped and nonhandicapped workers 
employed in service occupations. 

The VACG must be adn-iinistered by someone familiar with the 
worker's behavior in work settings. The evaliator simply marks 
with a check the answer that hesc describes the worker's 
behavior or skills. To compare the worker's VACG scores with 
scores expected of persons entering competitive employment, item 
scores are summarized within each of the subcategories, and 
score totals, are charted on the summary profile sheet. After 
trailing needs in a specific category have been identified, the 
evaluator checks the corresponding training goals in the 
curriculum guide. These goals can serve as the basis for 
development of individualized training plans. 

Summary . The use of work samples as vocational assessment 
tools is of little validity or utility for supported employment 
programs. Skill inventories, combined with job analysis 
information, are one ""/pe of vocational assessment which seems 
to have significant advantages over the work sample systems. 
The criteria used to evaluate an individual's performance on an 
inventory are derived from skil] requirements for actual jobs in 
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the community. Social validation techniques in which 
supervisors and co-workers are asked to document the accuracy 
and relevance of items and criteria in terms of the local job 
market are used in standardization. Information from the skill 
inventory along with job analysis data and situational 
assessment results can serve as the basis for developing 
training plans. 

Career Interests and Awareness 

The ai.sessment of the needs and interests of individuals 
with mental retardation has typically focused on developing 
measures to determine job preferences of this population. 
Originally such tests were considere ' of doubtful value, in that 
they attempted to assess interest in jobs that might not be 
available, either because of client's limited abilities or 
because of a restricted job market (Seltzer, 1984). However the 
growi , movement toward offering competitive work experiences 
at all levels of ability has tended to expand the range of job 
types open to persons with handicaps. Several measures of 
career interest have been developed for persons with handicaps. 
The Reading-Free Vocational Interest Inventory (RFVII) (Becker, 
1981). The RFVII is a picture format, non-reading interest 
inventory that assesses work preference at the ur skilled or 
semi-skilled levels in 11 areas--automotive , building trades, 
clerical, animal care, food service, patient care, ho^ "-iculture , 
housekeeping, personal service, laundry service, and materials 
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handling. Little training is necessary for the examiner. The 
examiner presents 55 triads of pictures from which the examinee 
selects his or her preferred activity in a forced-choice 
response forma**. The RFVII was standardized with students with 
mild mental retardation and learning disabilities at the high 
school level (N = 2,000) and on trainees at vocational training 
centers or employees of sheltered workshops (N = 2,000). The 
1981 edition has 10 norm groups, including LD. The sample was 
stratified on geographical location and socioeconomic status. 
Construct validity coefficients ranged from .00 to .82. No 
predictive validity attempts have been reported. Test^retest 
reliability over a two-week interval ranges from .70 to .80. 
Internal consistency coefficient equals .32. 

Several criticisms of the RFVII warrant consideration before 
it is considered for use. First, the use of a forced-choice 
format wi^h persons with severe handicaps has not been 
validated. The fact that males are pictured more frequently 
than females in work tasks has led to criticisms of sex bias. 
Finally, job types and response format may not be appropriate 
for more severely disabled individuals. These criticisms limit 
the use of RFVII in supporte<?. employment. 
Vocational Interest and Sophistication Assessment (VISA) 
(Parnicky, Kahn, & Burdett, 1971). VISA is a pictorial 
interest survey designed especially for persons with mental 
retardation. The test is administered individually and is hand 
scored. There are separate forms for males and females. The 
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male form provides int irest and knowledge scores in seven 
areas: garage, laundry, food service, maintenance, farms and 
grounds, materials handling, and industry. The female form 
provides interest and knowledge scores in four areas: business 
and clerical, housekeeping, food service and laundry, and 
sewing. Once again, sex bias may be an issue and should be 
considered before this instrument is used. Reliability and 
validity data are limited. 

CHOICE (Beziat, 1978). Based upon the pioneering work of John 
Holland (1959), CHOICE is intended for use with individuals with 
mild or moderate mental retardation. Despite this consideration, 
the inventory requires the client to make fairly sophisticated 
decisions regarding personal interests, aspirations, 
preferences, and estimations of self -competencies . Information 
is presented in the form of color slides, audiotapes, and 
drawings. The inventory is self -administered and self -paced. 
Although standardization was done with individuals in 
rehabilitation facilities and vocational evaluation units, the 
validity of this test for use with persons with moderate and 
severe disabilities has not been established. 

Summary . It is important to remember ^hat if career 
interest instruments are used, they should reflect career 
opportunities that are actually available to the individual, 
given his or her level of functioning and the kinds of jobs 
available in the community, job development efforts should be 
aimed at creating a broad range of employment options. Job 



standardized Assessment 

85 

matching efforts should take worker preference into account when 
selecting placement sites. Many people doubt the ability of 
persons with handicaps to have and express a career preference, 
but research findings suggest that matching a work3r with 
moderate retardation to a job on the basis of inventory 
responses appears to enhance job retention (Becker, Schull, & 
Campbell, 1981). 

Several measures of vocational interest have been developed 
specifically for persons with hardi^-aps, but unfortunately, most 
of these tests are useful only with persons with mild 
handicaps. In addition, psychometric flaws limit their use to 
exploration rather than decision making. Despite these 
limitations, they do appear to meet the usual requirements of 
interest tests, that is, a framework for the exploration of 
vocational interests, either to increase options or to narrow 
the range of possible choices (Hansen, 1984). For persons with 
more severe handicaps, a better option to obtain a good match 
between worker interest, ability, and job may be to pros/ide for 
a varied background of work experience upon which ai\ individual 
can base his or her preference or interest in a job type. 

Language 

Expressive and receptive language skills directly influence 
an individual's ability to '-''of it from verbal direction, the 
nature and quality of his c her social interaction, and his or 
her willingness to offer and seek assistance. Any or all of 
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these factors may contribute to success or failure in supported 
employment • Standardized tests of language may be useful to 
describe the characteristics of the target population or to 
obtain a quick assessment of the language abilities of new 
referrals. However, we know little ^bout the relationship of 
language scores to job performance. Standardized tests of 
language ability rarely contain job-specific items. These two 
factors limit the usefulness of standardized language tests for 
purposes of placement, program planning, and monitoring client 
progress . 

Because different wor}^ settings may have different technical 
and social language demands, it may be most meaningrul to assess 
language skills in the context in which they occur. Informal 
observation or structured situational assessment can be used to 
assess work-related language skills in this manner. In addition 
many adaptive behavior measures have a number of items that 
pertain to an individual's use of language in the context of 
daily living. 

Despite limited utility, tests of language are widely used in 
vocational prograjns. 

The Peabody Picture Vocabulary Test - Revised (PPVT-R) (Dunn & 
Dunn, 1981). The PPVT-R has replaced its earlier version, the 
PPVT, (Dunn, 1959) as the most commonly used measure of language 
for persons with severe handicaps (Pickett & Flynn, 1983). A 
nonverbal, multiple choice test, PPVT-R is designed to evaluate 
the receptive vocabulary ability of children and adults. The 
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vocational relevance of the items, however, is limited. Few of 
the items show any relationship to receptive vocabulary that may 
be necessary for vocational success. Reliability and validity 
are well documented. Administration procedures are clear. 
Scores for adults can be expressed as language ag-^^s (LAs) or as 
standard scores. The test allows for scores as low as 40. 

The PPVT-R is sometimes used incorrectly as a measure of 
general intelligence. Although the PPVT-R is useful for 
evaluating the ability of adults who are nonverbal cr physically 
handicapped, the single score and the sole reliance on receptive 
language as a measure of comprehension severely limit its 
substitution as a measure of intelligence* The PPVT-R should 
always be substantiated by other assessment information. 
Summary . The content and format of standardized tests of 
language bear little relationship to language demands of a work 
setting. Because different work settings have different 
technical and social language demands, it may be most meaningful 
to assess language skills in tne context in which they occur 
through the use of situational assessment or language subscales 
of tests of adaptive behavior. Information on an individual's 
language skills obtained in this way can be used in conjunction 
with job analysis data in job-matching activities to assess the 
congruence between amount and type of languare required on a 
particular job and a particular worker's language performance. 
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Academic Skills 

Although academic achievement of an adult with an IQ of less 
than 70 is almost by definition not greater than Grade 6 or MA 
11, achievement tests such as the WRAT and the PIAT are often 
used to assess clients with mental retardation. The usefulness 
of information gained in this manner is questionable • Norms are 
generally not available for adults with handicaps. More 
important, the relationship between academic skills as measured 
by these tests and job performance is unknown • what is probably 
most important to know in supported employment is the worker's 
functional academic level, that is, ability to read signs, 
write one's name, and manage time and money. These skills are 
better assessed in measures of adaptive behavior or by 
observation than by the use of standardized tests. 

Special Aptitude Tests 

A variety of aptitude tests have been utilized with adults 
with handicaps, including measures of fine and gross motor 
skills (e.g., Purdue Pegboard Test), visual discrimination 
(e.g., Minnesota Clerical Test), and spatial relations ( e.g., 
Minnesota Spatial Relations Test). Such measures are of 
questionable value for use with persons with severe handicaps 
(Rosen, Clark, & Kivitz, 1977). They can add little to the 
understanding of individual differences when the entire 
population of persons with mental retardation may score in the 
first percentile. In this sense norms serve only to classify 
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most subjects as unemployable. Even the development of more 
appropriate norms that are based on a population of persons with 
mental retardation is not much of an improvement. These 
isolated skills are not representative of the complexities of a 
real life employment setting (Menchetti et al., 1983) and should 
not be used to determine appropriateness of an individual for a 
particular job. 

Lif estyle/Consuitier Satisfaction 

This construct is especially important for evaluating the 
overall impact of supported employment programs on the lives of 
persons with handicaps. Therefore, measures of lifestyle/ 
consumer satisfaction are particularly useful in program 
evaluation. Areas to be measured include the degree of 
residential, work, and leisure integration, financial 
independence, extent of support network, presence of family and 
friendships, feelings of security and satisfaction with life, 
and degree of personal control. Some of these areas may be 
captured b^'^ measures of adaptive behavior, but others are not 
commonly found in any standardized measure. Many projects 
choose to develop informal questionnaires or semi-structured 
interviews to assess the level of satisfaction of their 
employees. A small number of standardized measures are 
available. 

Lifestyle Satisfaction Questionnaire (LSS) (Heal, Chadsey-Rusch, 
& Novak, 1982). The LSS consists of 29 items that are asked of 
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the client in an interview format to assess satisfaction with 
residence, comnunity, work, friends, and opportunities. Mean 
scores are available for a sample of 38 individuals with mental 
retardation. An acquiesence subscale makes it possible to 
correct satisfaction scores for acquiesence bias. Empirical 
data indicate this experimental version of the LSS has 
internally consistent subscales and good test-retest and 
interrater reliabilities. 

Quality of Life Questionnaire (Schalock, 1986). The Quality of 
Life Questionnaire is a 28-item instrument that can be completed 
by the examinee or by two staff members who are familiar with 
the examinee, items relate ro living conditions, personal 
control, social contact, financial independence, and job 
satisfaction, and are scored on a 3 -point scale. 

Summary . Despite the importance of lifestyle/consumer 
satisfaction as a construct in evaluating the impact of 
supported employment programs, few standardized instruments 
exist to measure this construct. If the instruments described 
here do not meet the evaluation needs of a project, project 
staff may consxaer using social validation techniques to design 
and validate an instrument. 

Conclusion 

The process of selecting, administering, and using 
information from standardized instruments in supported 
enployment is complicated, time consuming, and can be 
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unrewarding. To minimize the negative consequences of 
standardized assessment and to maximize the benefits, careful 
consider ition must be given to assessment instrumentation, 
process, and use. 

Although successful employment of an individual with 
handicaps requires a great deal of information, not all of that 
information is best obtained through the use of standardized 
assessment. Entry-level language skills, academic skills, and 
vocational skills may be best assessed in the employment setting 
through structured observation or situational assessment 
(Menchetti, Rusch, in press; Panscof ai , 1986). In this way, 
assessment can be sensitive to the interaction between skill 
level and job requisites, thus providing more relevant 
information for training and program planning. As rr.easured by 
standardized tests, changes in skill levels in these areas are 
less important indicators of program effectiveness than more 
global indicators such as "^ob retention and productivity. 

On the other hand, standardized intelligence tests can be 
very useful for describing the characteristics of the target 
population in terms of general ability. Tests of adaptive 
behavior provide information on social skills, daily living 
skills, level of inde^:endence, and level of integration for 
program planning, monitoring progress, and evaluating program 
impact. Information obtained from career interest inventories 
can assist job developers and job placement specialists in 
matching an employee with a job that is not only feasible at his 
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or her skill level, but is of interest to the employee. Because 
improving the quality of life for handicapped individuals should 
be the overriding concern of all human service progra-ns, 
measures of lifestyle satisfaction should be the primary 
yardstick by which we measure prcgram success. 

An example of -"i assessment plan for a supported employment 
program might be conducted as follows. Upon entry to the 
program, individuals without current (less than three years old) 
intelligence test scores would be given either a WAIS-R (for 
persons in the high moderate and mild range of functioning) or a 
Stanford-Binet: Form L-M (for persons in the low moderate and 
severe range of functioning). Adaptive behavior scores should 
be obtained for all individuals upon entry to the program. The 
Scales of Independent Behavior (SIB) or the Inventory for Client 
and Agency Planning (ICAP) axe two of the most easily 
administered and appropriate measures for this purpose. Also 
upon entry, a lif escyle/consumer satisfaction scale such as the 
Lifestyle Satisfaction Scale (LSS) could be administered to 
enable pre-post placement comparisons measuring the impact of 
supported employment upon quality of life. Information from a 
vocational skills inventory such as the VACG, from situational 
assessments, and from job analysis can be used to match an 
individual with a potential job or to assess the degree of fit 
between an individual's skills and the requisites of the job in 
which he or she is placed. These on-site assessments may need 
to be repeated at each new job placement. 
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Adaptive behavior and lifestyle/consumer satistaction measures 
can be readministered at one- or two-year intervals, depending 
upon program planning or program evaluation. 

Given the unique characteristics of the population served, 
special consideration must be givon to the selection, 
administration, and use of the results of standardized tests in 
supported work/employment. As these programs increase in 
popularity, the development of new measures may make the job 
easier. As always, we should be searching for ways to gain 
relevant information quickly and accurately irx order to 
prescribe and evaluate those services that best serve these 
individuals. 
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An Analysis of Select Psychometric Properties 
of the Vocational Assessment 
and Curriculum Guide 

Bruce M. Menchetti* and Frank R. ' usch 



Vocational assessment practices have a tremendous impact 
upon the lives of people with handicaps. In the course of 
seeking vocational services, these individuals may be asssesse<? 
by several agencies, each with its own instrumentation and 
purpose. Responding to federal legislation, local school 
districts provide an assessment of the interests, needs, ana 
abilities of vocational education students who have special 
needs. State vocational rehabilitation departments provide 
assessment services that determine an individual's potential for 
gainful employment. The U.S. Employment Service utilizes a 
battery of tests developed to match a person's work traits and 
aptitudes to specific occupational groups. With so many 
agencies providing such a variety of assessment services, one 
would expect the employment training opportunities available to 
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persons with handicaps to be unlimited. Unfortunately, this is 
not the case. Study after study investigating the post-school 
adjustment of adults with handicaps, especially those with 
mental retardation, have suggested that these individuals 
experience high rates of unemployment, receive much lower wages 
than people who are not disabled, and are often placed in 
programs that do not affectively promote movement into community 
employment (Hasazi, Preskill, Gordon, & Collins, in press; 
Mithaug, Horiuchi , & Fanning, 1985; Wehman, Kregel, & Seyfarth, 
in press; Will, 1984). 

As the initiation point of service delivery, assessment has 
played a key role in discouraging employment. Ir too many 
cases, vocational assessment has not been the means of 
identifying training opportunities, but instead has functioned 
to limit access to employment. Several reasons have been 
suggested for this paradox. Some have pointed out that many of 
the standardized measures of intelligence, achievement, and 
aptitude used for predicting successful vocational adjustment of 
persons with handicaps have never been validated for this 
purpose (Gold, 1973; Net:, 1966; Wolf ensberg^r , 1967). Others 
have suggested that the purpose of many vocational assessment 
programs, namely the prediction of general performance, may be 
unrelated to the more relevant goal of Mentifying specific 
training needs (Cobb, 1972; Halpern, Lehmann, Irvin, & Heiry, 
1982; Menchetti, Rusch, & Owens, 1983; Schalock & Karan, 1979). 
Whatever the reasons, traditional assessment services have never 
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fulfilled the purpose stated by the Vocational Evaluation and 
Work Adjustment Association (VEWAA). VEWAA has stated that any 
assessment effort must benefit both the service provider and the 
client by providing information that facilitates their joint 
development of a plan of action (Schneck, 1931). For many 
persons the result of traditional vocational assessment has been 
inaction and ineligibility. 

Some professionals have called for a major shift in the 
orientation of vocational assessment services for adolescents 
and adults with handic^.ps. For example, Menchetti, Rusch, and 
Owens (1983) suggested replacing vocational assessment 
procedures that embrace a psychometric, prediction orientation 
with assessment procedures that emphasize measurement of actual 
skills required for employment. This approach to assessment is 
referred to as "ecological analysis." Wehman, Renzaglia, and 
Bates (1985) have defined ecological analysis as a systematic 
approach to identifying skills that have a high priority for a 
person to learn. One instrument that includes skills that have 
been identified by employers is the Vocational Assessment and 
Curriculum Guide or VACG (Rusch, Schutz, Mithaug, Stewart, & 
Mar, 1982). Rusch, Schutz, and Agran (1982) surveyed employers 
in service and ' ight industries to determine t . skil required 
for entry-level jobs. The results of this research provided the 
basis for selecting items to include in the VACG. 

The VACG is a behavior rating sea* i designed to provide a 
measure of the vocational and social skills of persons with 
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handicaps. The VACG comprises eight domains, including 
attendance/endurance , independence , production, learning , 
behavior, communication, social skills, and self-help skills* 
There are 66 items on the VACG, each beginning with the phrase, 
"Does the worker," followed by a description of the behavior 
being assessed. Several alternative responses are provided, 
which indicate the level of the behavior displayed by the 
client. Raters are instructed to select the phrase that best 
describes the individual's current level of functioning. The 
VACG was designed to be used by classroom teachers, 
rehabilitation counselors, adult service providers, parents, and 
paraprof essionals to determine an individual's skill level in 
relation to standards suggested as important for success in 
service occupations, such as the food service industry or 
janitorial work, as well as light industrial occupations. The 
primary purpose of the instrument is to assist in the planning 
of instructional programs for individuals interested in 
competitive employment. 

This study attempted to investigate the psychometric 
properties oi the VACG in a manner consistent with technical 
standards. Specifically, this study examined several aspects of 
reliability and validity of the VACG using subjects with 
characteristics similar to those for whom the instrument was 
designed. Test-retest, internal consistency, and interrater 
reliability issues were addressed. Reliability data were 
collected for each VACG domain and, in some instances, the total 
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test score. The validity of the VACG for the purpose of 
ditf erentiating between individuals based oi- employment 
characteristics was addressed in four separate subject groups. 

Method 

Reliability 

Three aspects of the reliability of the VACG were 
addressed. Test-retest coefficients were computed to estimate 
the stability of the VACG scores. Internal consistency 
coefficients were computed to estimate the homogeneity of VACG 
domains and the total test. Finally^ interrater reliability was 
estimated by determining the agreement percentages obtained by 
separate observers using the instrument. 

Subjects . The subjects used to estimate the test-retest 
reliablity of the VACG were 98 mentally retarded and 
raultihandicapped individuals. These subjects were grouped 
according to their current employment status. Forty-seven 
handicapped persons employed exclusively in sheltered work 
settings, 34 individuals who had once worked in the competitive 
labor force, but were terminated and returned to a sheltered 
situation, and 17 handicapped workers employed in nonsheltered 
community jobs for periods of six months or longer constituted 
che sample for the test-retest procedure. 

The subject sample was made up of individuals whose primary 
diagnosis was moderate to severe mental retardation (IQ below 
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55). Many individuals in the sample also displayed secondary 
handicapping conditions such as orthopedic handicaps, emotional 
disturbance, and speech and language disorders. The subjects 
ranged in age from 18 to 57. There were an approximately equal 
number of males and females in the sample. 

Subjects who were not competitively employed at the time of 
the study were served by a local rehabilitation facility. This 
facility employed individuals in both a regular work program and 
a separate work activity center. Subjects were placed in one of 
these settings based on their individual productive croacity. 
Subjects who were competitively employed at the time of the 
study were working in a variety of food ser^^ice and janitorial 
jobs. These community employment settings included university 
dormitory kitchens, restaurants, hotels, and other typical 
service occupation sites. 

A pooled sample of all the ratings in both the reliability 
and validity phases of the study was used to calculate the 
internal consistency coefficients. This resulted in scores from 
364 separate administrations being used to calculate the 
split-half reliabilities. A subsample of six subjects, ranging 
in age from 21 to 26, was used to calculate the interobserver 
percentages. Five male subjects and one female subject 
participated in this aspect of the reliability procedure. 
Generally, the procedures used in the study represented the 
traditional psychometric techniques most likely to be used to 
establish the reliability of instruments such as the VACG. 
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Data collection procedure . Subjects were rated on two 
separate occasions by raters who were familiar with their 
behavior. Subjects were assessed by individuals with whom they 
worked, or who observed them at work in their jobs. These 
raters included workshop supervisors, workshop trainers, job 
coaches, and work supervisors in community work settings. These 
raters represented individuals who would be prospective users of 
the ^^ACG. Subjects working in the sheltered work setting were 
rated by their workshop supervisors; competitively employed 
subjects were rated by job coaches or their immediate work 
supervisors. 

The raters who participated in the reliability procedure 
were asked to read the administration instructions provided with 
each VACG ana to complete the instrument. All raters had 
previously used the VACG to rate workers and were familiar with 
the administration of the instrument. This administration 
procedure parallels the first-person assessment procedure 
suggested for traditional behavior rating instruments such as 
the Adaptive Behavior Scale {Nihira, Foster, Shellhaas, & 
Leland, 1975). Each rater was assigned from one to five 
subjects to assess. After completing the first rating with the 
VACG, the same raters were asked to complete another inventory 
for the same subjects. The time period between the first and 
second ratings was approximately two weeks. 

To determine the ihternal consistency of the VACG, 
coefficients of equivalence were computed. Coefficient alpha 
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(Cronbach^ 1951) was used to obtain coefficients for each VACG 
domain score and for the total test score. The alpha 
coefficients are generally considered good approximations of 
consistency between items not scorable using the dichotomous 
zero-or-one procedure. The alpha coefficients represented the 
average of all possible split-half coefficients. 

To estimate the interrater reliability of the VACG^ six 
subjects were rated by two separate observers on the same 
measurement occasion. The raters were employed as job coaches 
by the rehabilitation facility. All six subjects were trained 
on a mobile work crew performing custodial work. Their behavior 
was assessed by the raters as the subjects cleaned one of the 
contract sites. Each subject was rated on the same day in order 
to keep the raters' experience and knowledge of the subjects 
consistent. A percentage of interobserver agreement was 
obtained for each of the eight domains and the total test score 
for each of the six subjects. This resulted in 54 interobserver 
agreement percentages. The percentage of interobserver 
agreement was determined by dividing the lower score by the 
higher score obtained by the raters for each VACG domain. 

Validity 

In order to determine whether the VACG was useful in 
determining appropriate instructional services for individuals 
with mental retardation interested in competitive employment, 
the following validation procedure was designed. The procedure 
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attempted to determine if the VACG could differentiate between 
groups of individuals with various levels of work experience. 
The rationale behind this approach was that if the VACG was to 
be useful in the specification of instructional objectives for 
persons with handicaps, then the individual's score on the 
instru^^ent should be related to a specified level of work 
performance. With knowledge such as this, VACG users could 
prescribe valid instructional goals and objectives to facilitate 
advancement to the next level of work performance. 

Subjects . There were 131 subjects in the validation 
sample. These subjects represented four worker groups with 
different levels of work experience. Fifty-nine workers in the 
sample (Group A) had only sheltered work experience. These 
individuals had never worked in the competitive labor force and 
were employed in a variety of sheltered work settings including 
a regular work program, a work adjustment program, and a work 
activity center. The majority of subjects in Group A, however, 
were employed in the work activity centrr at the time of the 
study. Twenty- six subjects (Group B) who were employed in 
sheltered work settings at the time of the study had once had 
competitive employment experience, but had been fired. These 
individuals were also employed in a variety of sheltered work 
programs with the m^^iority working in a work adjustment 
program. Nineteen individuals (Group C) were disabled workers 
employed in a variety of competitive employment settings in the 
community. All of these individuals had previous sheltered work 



ERLC 



I»8 



Vocational Assessment 
108 

experience, had participated in a competitive employment 
training program, and had been working successfully wich various 
degrees of support for six months or longer at the ^-ime of the 
study. Finally, 27 subjects (Group D) in the validation study 
were not disabled, had never been placed in a sheltered 
workshop, and were employed in a variety of competitive 
employment settings in the community. 

The subjects in Groups A, B, and C were individuals v^ith a 
primary handicap of mental retardation* Some of the subjects in 
these groups had secondary handicapping conditions such as 
speech and language disorders or orthopedic disabilities. 

The nonhandicapped subjects (Group D) were selected for the 
study based on several factors. First, these individuals were 
selected on the basis of their occupation. The subjects in 
Group D were employed in a variety of positions in the food 
service and janitorial industries, such as kitchen helper, 
kitchen laborer, and maid. A second selection criterion for 
Group D subjects was that these individuals did not exhibit or 
report any disability at the time of the study. Finally, Group 
D subjects were selected on the basis of cheir work 
performance. These subjects were "average" workers, that is, 
individuals who did not consistently fall either above or below 
their employers' expectations for work performance. 

Data collection procedure . The raters used in the 
validation phase of the study were prospective users of the 
VACG, including workshop supervisors, workshop trainers, job 
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coaches, and job supervisors in the competit '.ve employment 
settings. The subjects were rated by individuals with whom they 
worked or who observed them at work in their individual 
employment situation. All raters were familiar with the 
behavior of the subjects assigned to them. 

Raters who participated in the validation study were 
instructed to complete the VACG for assigned subjects in a 
manner siirdlar to the method described in the reliability 
procedure. 

The settings in which the subjects' behaviors were rated in 
the validation procedure were similar to those used in the 
reliability phase of the study. Group A and B subjects were 
rated in various sheltered settings, including regular work 
programs, work adjustment programs, and work activities 
centers. Subjects in Groups C and D were rated in a variety of 
competitive job sites in the community such as kitchens, 
restaurants, hotels, and other service industry sites 

Results 

Test-Retest Reliability of the VACG 

Coefficients of stability were computed to estimate the 
stability of each of the VACG domains for each subject group to 
determine if domain scores would remain equally stable for 
workers with different employment experiences. 

The test-retest coefficients and standard errors of 
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measurement for the first subject group are summarized in Table 
1. The test-retest coefficients for subjects having only 
sheltered work experience ranged from slightly over .76 in the 
learning domain to over .95 in the communication skills area. 
The median of the correlation coefficients for this group was 
approximately .88. 

The stability coefficients for the second subject group are 
reported in Table 2. These subjects represented persons with 
mental retardation who had some competitive employment 
experience, had exited the labor force, and were working in a 
sheltered work program at the time of the study. These data 
indicated a range of reliability from .71 in attendance/ 
endurance to greater than .96 in the communication skill 
domain. The median for the test-retest coefficients for this 
group was .88. 

The test-retest coefficients for the final group, those 
subjects with mental retardation who had been working in 
competitive jobs for periods of longer than six months, are 
reported in Table 3. The range of reliability coefficients was 
-.69 in attendance/endurance to approximately .91 in 
communication skills. The median of the test-retest 
coefficients was .84 for this group of subjects. 

Internal Consistency of the VACG 

Coefficients of equivalence were computed to estimate the 
internal consistency of the VACG. Coefficient alpha (Cronbach, 
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Table 1 

Test Retest Relieibility CoeffiLi«nLs for Subjects Having Only ShelLered 
Work Experience (n = 47) 



First Second 
Administration Administration 



Domain 


Mean 


SD 


Mean 


SD 


SEM 


r 


Attendance/ endurance 


7,83 


2.06 


8.11 


2.08 


0.76 


.862 


Independenc 


14.72 


3.98 


14.28 


3.98 


1.85 


.783 


Production 


A. 66 


3.39 


4.70 


3.26 


1.09 


.897 


Learning 


7.36 


3.34 


7.34 


3.22 


1.62 


.766 


Behavior 


13.51 


6.21 


12.30 


6.60 


2.05 


.891 


Communication skills 


17.30 


8.93 


16.32 


8.85 


1.89 


.955 


Social skills 


2.40 


1.51 


2.32 


1.40 


0.63 


.824 


Self-help skills 


4.91 


2.09 


4.98 


2.20 


0.65 


.902 


Mean 










1.318 




Median 












.877 
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Table 2 

Test-Retest Reliability Coefficients for Sheltered Worksnop Subjects With 
Previous Competitive Employment Expetxeace (n = 34) 



First Second 
Administration Administration 



Domain 


Mean 


SD 


Mean 


SD 


SEM 


r 


Attendance/endurance 


7.59 


2. 71 


8.09 


9 fifi 

^ m \J\J 


1 AS 




Independence 


16.06 


4.70 


16.88 


4.22 


2.36 


.747 


Production 


9.65 


3.72 


9.76 


3.90 


1.33 


.872 


Learning 


9.56 


2.82 


9.94 


2.89 


0.86 


.906 


Behavior 


16.86 


5.98 


17.38 


5.70 


2.03 


.885 


Communication skills 


26.35 


7.97 


26.79 


7.62 


1.53 


.963 


Social skills 


3.32 


1.25 


3.53 


1.06 


0.42 


.886 


Self-help skills 


5.71 


1.84 


5.76 


2.00 


0.88 


.772 


Mean 










1.358 




Median 












.879 
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1951) was used to obtain correlations . ' all possible split-half 
reliabilities • Table 4 summarizes the internal consistency 
data. The coefficients obtained ranged from a low of .59 in 
attendance/endurance to a high of .91 in communication skills. 
The median of the split-half coefficients was .78. The 
coefficient obtained for the total VACG score was approximately 
• 95. 

Interobserver Reliability of the VACG 

The interobserver agreement percentages obtained for VACG 
domain scores ranged from 90 to 100. The agreement percentages 
obtained for each VACG domain were: attendance/endurance = 98; 
independence = 90; production = 94; learning = 91; behavior = 
94; communication skills = 97; social skills = 100; and 
self-help skills = 100. The average agreement percentage for 
the total test score was 96. 

Validity of the VACG 

Domain score averages for each of the VACG domains are 
depicted in Figure 1 for the four subject groups used in the 
validation procedure. The pattern of the difference between 
groups suggests that subjects with less restrictive work 
placements who have successful competitive employment experience 
tend to score higher on VACG domains than do subjects placed in 
more restrictive, sheltered settings. The two subject groups 
with successful competitive employment experiences (Groups C and 

9^. IPi 



Table 3 

Test-Retest Reliability Coetticients for Competitively Employed Subjects 
(n = 17) 



First Second 
Administration Administration 



Domain 


Mean 


SD 


Mean 


SD 


SEM 


r 




Q A 1 


0 1 / 
Z. 14 


12. 88 


12. 84 


2. 78 


-.686 


Independence 


18.71 


1.87 


18.71 


2.02 


0.87 


.785 


Production 


10.94 


3.08 


11.35 


3.10 


1.21 


.846 


Learning 


8.00 


2.50 


8.47 


2.23 


0.95 


.857 


Behavior 


19.76 


2.53 


19.88 


2.49 


0.81 


.900 


Coramun icat ion ski 1 Is 


27.59 


5.29 


28.00 


4.55 


1.60 


.909 


Social skills 


3.71 


0.57 


3.59 


0.77 


0.26 


.794 


Self-help skills 


6.41 


1 . 3/ 


6.12 


1.49 


0.55 


.838 


Mean 










1.129 




Median 












.842 
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Table ^ 



Alpha Coefficients for the Vocational Assessment and Curriculum Guide 



Domain 


Number of Items 


Mean 


SD 


r 


Attendance/endurance 


3 


7 


.13 


2 


.78 


.588 


Independence 


9 


15 


.15 


3 


.83 


.675 


Production 


7 


7 


.17 


A 


.55 


.758 


Learning 


6 


7 


91 


3 


.07 


.709 


Behsvior 


9 


15 


83 


5 


.68 


.810 


Conmunication skills 


21 


21 


7A 


9 


.08 


.906 


Social skills 


A 


3 


OA 


1 


.Al 


.81A 


Self-help skills 


7 


5 


51 


1 


.89 


.793 


Total 


66 


U 


29 


25 


.32 


.9A6 
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D) consistently obtained the highest or second highest average 
domain scores, whereas the two subject groups who ;;ere working 
in sheltered settings (Groups A and B) consistently scored 
lowest on VACG domains. The two exceptions to this pattern 
occurred in the independence and learning domains. In the 
independence domain, subjects who were fired and returned to a 
sheltered work setting averaged slightly higher than nondisabled 
workers, but not as high as competitively employed persons with 
mental retardation. In the learning domain. Group B subjects 
who had some competitive employment experience but had returned 
to a sheltered setting scored higher than Group C subjects 
(competitively employed workers who were retarded), but not as 
high as the nondisabled workers. These exceptions might be due 
to the effects of previous competitive experience on the 
perceptions of raters assessing Group B subjects. 

In order to determine whether the group differences depicted 
in Figure 1 were statistically significant, several multivariate 
tests were applied to the data. A Wilks lambda criterion 
(.4x4), a multivariate F-ratio (Pao = 10.0254), and a 
Rawley-Hotelling trace (Tau = 1.1807) were obtained. The 
multivariate F-ratio and Rawley-Hotellinq trace tests were both 
significant beyond the .001 probability level. 

To evaluate whether or not the group differences were 
significant for specific VACG domains, several univariate 
F-tests were utilized. In this procedure, univariate analyses 
of variance were performed on the data obtained for each of the 
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Figure 1 Domain score averages for each of the Vocational 
Assessment and Curriculum Guide domains 
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Table 5 



Results of Univariate Analysis of Variance for VACG Domains 



Domain 


MS-treatment 


MS 


-error 


F-ratio 


Probability 


Attendance/ endurance 


103.2935 


17 


. 7726 


5.8119 


.0011 


Independence 


181 


.6950 


lA 


.9765 


12. 1320 


.0000 


Production 


637 


.9179 


12 


.6872 


50.2806 


.0000 


Learning 


103 


A56A 


8 


9568 


11.5A82 


.0000 


Behavior 


510 


7368 


32 


5077 


15.7113 


.0000 


Communication skills 


2A39 


650A 


59 


A721 


A1.0218 


.OOCO 


Social skills 


26 


A317 


1. 


5018 


17.5995 


.0000 


Self-help skills 


39. 


2A53 


3. 


A587 


11.3A70 


.0000 



DF (treatment) = 3 
DF (error) = 240 
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Table 6 

Differences among Means Evaluated against HSD for Each VACG Domain 






Attendance/endurance (HSD = 2. 91 A) 












X2 Xj 






^2 = 7.79 - 0.03 0.8A 


3.18* 




^1 = 7.82 - 0.81 


3.15* 




H - 8.63 


2.34 




^3 = 10.97 


- 




*p < .05 






Independence (HSD = 2.675) 












xi x^ X2 


X3 




^1 = 14. A2 - 1.58 2. OA 


A.4A* 




= 16.00 - 0.A6 


2 . 86* 




^2 = 16. A6 


2. AO 




^3 = 18.86 


- 




*p < .05 






Production (HSD = 2.A52) 














'^A 




'^l = A. 86 - A. 90* 6.06* 


6.51* 




^2-9.76 - 1.16 


1.61 




= 10.92 


0.A5 




XA = 11.37 


- 




*p < .05 
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Table 6 (con't) 



Learning (HSD = 2.069) 



xi = 7.27 
^3 = 8.24 
^2 = 9.70 

H ^ 9.74 

''p < .05 



0.97 



2.43 
1. 46 



2.47 
1.50 
0.04 



Behavior (HSD = 3.942) 



^1 - 13.17 
^2 - 17.00 
= 17.67 

^2 - 19.67 
-'p < .05 



^^2 



3.83 



4.50 
0.67 



6.59-^ 

2.76 

2.09 



Communication skills (HSD = 5.331) 



xi - 16.79 
^2 = 26.42 
^3 = 27.57 
H = 30.52 
*p < .05 



9.63* 



10.78^' 
1.15 



13.73* 
4. in 
2.95 



Table 6 (con't) 



Social Skills (BSD = :.8472) 



H - 2.A6 
X2 = 3.45 
H - 3.-9 
X3 - 3.68 
*p < .05 



0.99* 



1 . 13* 
O.IA 



1.22* 

0.23 

0.09 



Self-help skills (HSD = 1.286) 



''I = 4.89 

X2 = 5.75 

^3 = 6.27 

H = 6.89 
*p < .05 



0.86 



1.38* 
0.52 



2.00* 

1.14 

0.62 
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eight domains. Table 5 lists the F-ratios obtained for each 
VACG domain. The probability levels for each F-ratio obtained 
were equal to or less than .001, which suggests that the 
observed group differences were statistically significant for 
each VACG domain. 

Tukey's honestly significant difference tect (HSD) was 
performed to evaluate the differences among group ^neans. The Q 
statistic at the .05 probability level with infinite degrees of 
freedom was used to calculate the HSD value. The significant 
differences obtained are summarized in Table 6. 

Attendance/endurance . The competitively employed subject 
group comprising workers who were handicapped scored 
significantly higher than both sheltered work groups. There 
were no other significant differences among group means. 

Independence . The competitively employed, handicapped 
worker group average was significantly higher than that of 
disabled workers with no competitive employment work history and 
than the average obtained by nondisabled workers, which suggests 
that competitively employed workers with handicaps are perceived 
as being highly independent. This finding may be an artifact of 
rater bias, because many subjects in the high group were rated 
by their trainers, who have a vested interest in thei^- success. 
The sheltered employmenr grqdfc^^ho had previous competitive 
experience might have been rated high for the same reason, that 
is, rater bias. Also, raters in the sheltered setting may have 
inflated their ratings because of the perception that these 



Q individuals were high functioning. 

ERIC 133 



Vocational Assessment 

123 

Production. In the Production domain, workers with 
handicaps who had no comperitive employment experience were 
rated significantly lower than any other group. There were no 
other differences among group means. This finding may suggest 
that the VACG fails to discriminate the productivity of persons 
having some competitive work experience. 

Learning . In the Learning area, workers with handicaps w*ho 
had no competj*^ive employment experience obtained significantly 
lower scores than individuals who had returned to a sheltered 
setting from the competitive labor force and J!rom nonhandicapped 
workers. There was no significant difference, however, between 
the scores obtained by competitively employed workers with 
handicaps and either sheltered work groups. 

Behavior . Both disabled and nondisabled competitively 
employed subject groups scored significantly higher than 
sheltered workshop employees wit^ no competitive employment 
experience. This finding suggests that Behavior is a VACG 
domain that discriminates well between persons with no community 
work experience and persons who are competitively employed. 

Communication skills . In the Communication Skills domain, 
all subject groups obtained significantly higher scores than the 
sheltered work group with no competitive work experience. There 
were no significant differences between any other groups, 
however. 

Social skills . In the Social Skills domain, the pattern of 
significant differences among group means was the same as 
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that obtained in the Communication Skills area; that is, all 
subject groups were rated higher than sheltered workshop 
subjects with no competitive employment experience. 

Self-help skills . Both disabled and nondisabled 
competitively employed subjects were rated significantly higher 
than sheltered workshop employees with no competitive work 
history. In the Self-help domain there were no other 
significant group differences. 

On all but one VACG domain, namely the Learning area, there 
were significant differences between the average scores obtained 
by competitively employed workers with handicaps and sheltered 
workshop employees with no competitive employment history. This 
pattern suggests that, with the exception of the Learning 
domain, VACG ratings can be used to make valid discriminations 
between individuals in sheltered workshops seeking their first 
competitive employment placement and successful disabled 
employees in competitive settings. 

uiscussion 

This study was an attempt to apply classical psychometric 
procedures to the problem of establishing the technical adequacy 
of the Vocational Assessment and Curriculum Guide (VACG). The 
VACG, which was designed to facilitate efforts in planning 
programs for individuals with handicaps interested in 
competitive employment, is a criterion-referenced behavior 
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rating scale. More significantly, the VACG represents a new 
approach to vocational assessment, one based upon an analysis of 
the employment ecology. Previously, the lack of evidence 
supporting the technical adequacy of instruments with an 
ecological analysis orientation has been a major impediment to 
the acceptance of this approach into assessment practice. 

The appropriate procedure for judging the technical features 
of criterion-referenced instruments such as the VACG has been 
discussed in the literature. For example, Anastasi (1976) 
suggested that reliability procedures appropriate for 
norm-referenced instruments may not be appropriate for 
criterion-referenced measures. It is the belief of these 
authors that until a .suitable set of empirically based standards 
are developed for 'criterion-referenced instruments, these 
measures should be evaluated by mean,*-" of techniques accepted for 
norm-referenced inb^ruments , even though these techniques 
underestimate the techn: jal soundness of criterion-referenced 
instruments. For ecological analysis to gain d foothold in the 
assessment establishment, however, authors of instruments 
with this orientation may have to adopt the traditional approach 
to establishing the technical adequacy of their measures. 

The results of this study support the staDility of scores 
obtained with the VACG. Test-retest or stability coefficients 
wore found to fall within acceptable ranges for behavior-rating 
scales. Furthermore, this study reportea stability estimates 
for various groups of subjects who represented persons for whom 
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the VACG was designed. These estimates suggested that VACG 
domain score" were equally stable across persons having various 
employment characteristics. Stability of measurement is a 
desirable characteristic of any rating scale because it assures 
potential users that scores represent a lasting quality of the 
individual being rated (Cronbach, 1960). The evidence of score 
stability presented in this study will allow users of the VACG 
to have confidence that the scores they obtain will not change 
substantially over a short period of time. This, of course, 
will permit teachers, rehabilitation counselors, and other users 
of the VACG to plan vocational training programs with the 
knowledge that the plan has been based on an assessment of 
stable vocational and social skills. 

This study also presented evidence related to the 
consistency of the VACG scores. Data were presented for 
estimates of internal consistency as calculated by alpha 
coefficients for each VACG domain and the total test score. The 
estimates were in a generally acceptable range for this 
psychometric criterion. '. internal consistency estimates 
obtained in this study suggested that the VACG domain scores ana 
the total test score represented measures of homogeneous 
behavior. In ocher words, a person's rating on individual items 
of the VACG appears to relate to ooth the domain score and the 
total ter.t score. For prospective users of the VACG, the 
internal consistency findings of this study have practical 
significance. Given the results of this study, VACG users can 
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be assured that domains measure related behaviors. Internal 
consistency data provide evidence supporting the overall 
technical adequacy of the VACG. 

Another aspect of the reliability of the VACG examined in 
this study was interrater agreemeiit. This aspect of the 
reliability of measurement is the most frequently reported index 
for behavioral assessment instruments. The data presented here 
suggested that, in terms of interrater agreement, the VACG 
provides a highly reliable measure of the vocational and social 
skills of individuals with handicaps. Bates and Hanson (1983) 
have pointed out that the effectiveness of making data-based 
decisions, such as planning a program of vocational training, is 
riependent upon measurement reliability. Based on the data 
obtained in this study, it would appear that the VACG is a 
useful tool in making decisions about appropricite vocational 
training programs. 

The reliability data obtained in this study are also useful 
in comparing the VACG to other behavior rating scales that 
measure social and vocational skills. These comparisons can be 
useful in making judgments about the relative accuracy of the 
VACG. din, Levine, and Elzey (1977) reported test-retest 
coefficients ranging from .88 to .97 for the domains of the 
Cain-Levine Social Competency Scaie. These authors also 
reported an interrater reliability coefficient of .94 for their 
instrument. Halpern, Raffeld, Irvin, and Link (1975) reported 
test-retest and internal consistency coefficients for the Social 
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and Prevocational Information Battery. The test-retest 
coefficients for the nine domains of this instruiTient ranged from 
.70 to .79 for a junior high school sample and from .62 to .78 
for senior high subjects. Halpern et al. (1975) reported 
Kuder-Richardson coefficients of internal consistency ranging 
from .78 to .82 for the Social and Prevocational Information 
Battery. Finally, Nihira, Foster, Shellaas, and Leland (1975) 
reported interrater reliability coefficients of .71 to .91 for 
Part ^ne domains of the AAMD Adaptive Behavior Scale. 
Coefficients for Part Two domains ranged from .37 to .70. The 
reliability data obtained in this study of the VACG compare 
favorably with the results reported in these studies of similar 
psychometric properties of vocational rating scales. 

The results of the validation procedure suggested that 
domain scores of the VACG differentiated between groups of 
individuals with diverse employment characteristics. The domain 
score averages obtained by four riuoject groups with different 
vocational placements and work histories were significantly 
different when tested with several multivariate statistical 
procedures. Moreover, step-down, univariate analyses of 
variance were statistically significant for each VACG domain. 
Many of the observed differences were in a pattern in which 
subjects in the most restrictive, sheltered placements scored 
lowest and subjects in competitive employment sfi*-*-ings scored 
highest. Tukey's honestly significant difference tests 
suggested that, with the exception of the Learning domain, VACG 
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scores can be used to make valid discriminations between 
individuals seeking their first competitive job and successful 
disabled employees. 

Many of the observed differences between groups in the 
validation procedure are supporced by the literature comparing 
work performance of disabled and nondisabled individuals. Two 
studies of work performance were conducted at du Pont de 

Nemours and Company (E.I. du Pont de Nemours, 1982; Sears, 
1975). These studies surveyed 2,745 employees with a wide range 
of disabilities such as blindness, orthopedic problems, mental 
retardation, hearing impairi *,ts, and epilepsy. The findings 
provide a possible explanation for some of the differences that 
weie found between the competitively employed subjects in this 
study. For example, E.I. du Pont de Nemours reported that, in 
1981, the attendr^ce records of disabled and nondisabled 
employees were rable. This finding ray explain the higher 

ratings obtained . this study by the competitively employed 
subjects with handicc.ps in the attendance/endurance domain. 
Raters may have scored these individuals higher on attendance- 
related items because they met or exceeded the raters' 
expectr.tions for dependability. Other studies have suggested 
that similar employees have average or better than average 
attendance records (The President's Committee on Employment of 
the Handicapped, 1981). An earlier du Pont study suggested that 
79% of workers with handicaps were rated similarly or better 
than the total work force on measures of attendance (Wolfe, 
1973) • 
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Another VACG domain in which disabled, competitively 
employed subjects with handicaps were rated higher than their 
nondisabled counterparts was the independence area. Again, data 
from studies investigating the work performance of employees 
with handicaps in the competitive labor force suggested that 
these individuals may perform better than nondisabled 
employees. The du Pont studies reported that the disabled 
employees improved their performance of job duties trom 91% to 
92% over the survey years as compared to 91% for nondisabled 
workers (Sears, 1975; E.I. du Pont de Nemours, 1981). In a 
review of the social and cost factors involved in employing 
persons with handicaps, Weisgerber, Dahl, and Appleby ( J81) 
cited a joint National Association of Manuf acturers/U. S. Chc'onber 
of Commerce study which surveyed 279 businesses and found that 
approximately 90 % of the employers reported no cost increase as 
a result of hiring employees with handicaps, rated their job 
performance as good as other workers, indicated their attendance 
was better than nondisabled workers, and felt persons who were 
disabled had better work habits and safety records than 
nonhandicapped workers. Weisgerber et al. (1981) concluded 
their discussion of the work performance of persons with 
handicaps by stating that the findings are conclusive that it is 
good business practice to hire persons with handicaps. 

The literature has suggested that workers with handicaps may 
be better than nondisabled workers in areas such as attendance, 
work habits, safety, and job performance. The ratings of 
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competitively employed workers with handicaps on the VACG would 
support these findings. In the VACG validation, the 
competitively employed handicapped workers were rated higher 
than any other subject group in the attendance/endurance, 
independence, behavior, and social skills domains. These 
domains contain items measuring work atteiidance, safety, 
appropriate work habits such as completing assigned tasks, and 
work-related social skills including asking supervisors or 
co-workers for help. 

The recent emphasis on improving employment training options 
for persons with handicaps will have an impact on vocational 
assessment. As more training alternatives are developed, 
assessment procedures that have a direct relationship to 
curriculum planning and training will be needed. The ecological 
analysis approach, characterized by instruments such as the VACG 
that systematically identify survival skills, may fill this 
need. However, before selecting any instrument for a vocational 
assessment program, consumers must consider evidence of its 
technical adequacy. The data presented in this study will allow 
persons interested in the VACG to evaluate its appropriateness 
as a measure of the social and vocational skills of persons with 
handicaps who are interested in competitive employment in the 
service occupations. 
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ISEP 
Funded Projects 



Date/ Time 

Staff Facility and Address 



Contact Person 



Region & 
Agency 



6/16 Aid to Retarded Citizens Inc. 

9:00 2719 S. 11th St. 

DW Springfield, IL 62703 



6/12 Ctr. for Disabled Student Serv. 
9:00 Chicago City-Wide College 
JST 226 W. Jackson, 6th Floor 

Chicago, IL 60606 



6/26 The Center for Rehabilitation 
9:00 and Training of the Disabled 

JST 6610 N. Clark St. 

Chicago, IL 60626 



*6/3 Chicago Assoc. for Retarded 
9:00 Citizens 

OW/JT 8562 S. Vincennes 
Chi^igo, IL 60620 

6/15 Comprehensive Services 

9:00 Box 428 

JST Mt. Vernon, IL 62864 



6/12 Developmental Services Ctr. 
9:00 l6vH n. Bradley Avenue 
JT Champaign, IL 61821 



6/22 Effingham Association for 
9:00 Retarded Citizens 

DW 618 W. Main St. 

Teutopolis, IL 62467 

6/9 El Valor Corporation 

9:00 1850 W. 21st St. 

JT Chicago, a 60608 



6/19 Employer Resource Service/Arrise 
9:00 9790 Allen Street 
DW Rosemont, IL 60018 



Mark Schneider 
Patrick Wear, II 
Executive Director 
(217-789-2560) 



Dan Woodyatt 
Sally Vernon 
Director 
(312-641-2595) 

Carol Woodworth 
(312-929-8200) 
Cathy Lorber, Ph.D. 
Stuart G. Ferst 
Executive Director 
(312-973-790:)) 



3 

DMH/DD 
(411) 



5B 
DMH/DD 
(370) 



5A 
DMH/DD 
(403) 
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Robert Lewis 5B 

Carolyn B. Thompson DMH/DD 
Dir. of Placement Services (401) 
(312-651-3720) 

Sharon Smith 4 

John Metcalf DMH/DD 

Rehab. Director (231) 
(618-242-7300) 

Carole Powers 3 

Rick Krandel DMH/DD 

Dale Morrisey (422) 
Chief Executive Officer 
(217-356-9176) 

Michael Poe 4 

Michael W. Fortner DMH/DD 

Executive Director (475) 
(217-857-3186) 

Nina Martino 5B 

Gloria Morales Curtin DMH/DD 

Dir. of Adult Services (539) 
(312-666-4511) 

Rosalie Mattern 2 

James Johnson DMH/DD 

Executive Director (363) 
(312-771.2945) 
(312-671-5877 



(9-10-87) 



Date/Time 

Staff Facility and Address 
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Contact Person 



Region & 
Agency 



6/16 Franklin-Williamson Workshop 

9:00 902 W. Main St. 

JST West Frankfort, IL 62896 



6/15 Happiday Centers, Inc. 

9:00 1005 H. End Ave. 

JT Chicago Heights, IL 60411 



6/22 Iroquois ARC 

9:00 Box 324 

JT Watseka, IL 60970 



6/16 Job Resources for the Disabled 
9:00 3140 N. Cambridge 
JT Chicago, IL 60657 



*6/23 Lake County Society for 
9:00 Human Development 

OW 3441 Sheridan Road 
Zion, IL 60099 

*6/2 The Lambs, Inc. 

9:00 P.O. Box 520 

JST/JT Libertyville, IL 60048 



6/30 McDonough Co. Rehab. Ctr. 
9:00 900 S. Deer Road 
JST Macomb, IL 61422 



6/30 North Shore Association 
9:00 for the Retarded 

JT 7855 K Gross Point Road 
Skokie, IL 60077 

6/18 Occupational Development Ctr. 

9:00 400 H. East St. 

JST Bloomington, IL 61701 



Je^'f Horton 1 

Gail Kear DORS 

Executive Director (234) 
(618-983-5421) 
(618-937-6483) 

Diane Alexander 6 

Jim Kompik DMH/DD 

Patrick O'Brien (570) 
Executive Director 
(312-755-8030) 

Roberta loder 3 

Louellen Strong DMH/DD 

Executive Director (140) 
(815-432-5288) 
(815-432-6191) SEP Office 

Andy Tayaka 5 

Ms. Michael Rooney DORS 

Executive Director (337) 
(312-327-4412) 

Linda Milz 2 

Randy R. Ross DMH/DD 

Executive Director (419) 
(312-872-1700) 

Julie Metzger 2 

Gerald Y. Friedman DMH/DD 

Executive Director (391) 
(312-362-4636 or 361-3320) 

Mary Fran 1 

Jim Starnes DMH/DD 

President (364) 
(309-837-4876) 

Pat Baer 2 

Gerald Gully DMH/DD 

Executive Director (422) 
(312-869-5515) 

Paula Knutson 3 

Lyn Hyrska DMH/DD 

Tonm Foss (451) 
Executive Director 
(309-828-7324) 
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Oate/Tlme 

Staff Facilitw and Address 



6/9 Open Door Rehab. Ctr. 

9:00 208 Beaver Street 
DW Yorkville, IL 60560 



6/8 Opportunity House, Inc. 

9:00 202 Lucas St. 

JST P.O. Box 301 

Sycamore, IL 60178 

6/10 Orchard Village 

9:00 7670 Marmora 

JST Skokle, IL 60077 



6/18 Park Lawn 

9:00 5340 W. 111th St. 

JT Oak Lawn, IL 6C453 



6/8 Ray Graham Association 

10:00 420 W. Madison St. 
JT Elmhurst, IL 60126 



6/10 Seguin Services Inc. 
10:00 3145 S. 55th Avenue 
DW Cicero, IL 60650 



6/11 Special Education Parents 

9:00 Alliance 

JST 305 22nd St., Suite K-164 
Glen Ellyn, IL 60137 

*6/l Spectrum Vocatioml Serv. 

9:00 2302 Wisconsin 

DW Downers Grove, IL 60515 



6/17 Victor C. Neumann Assoc. 
9:00 2354 N. Milwaukee 
OW/JST Chicago, IL 60647 



Contact Person 



Region & 
Agency 



Steve Strieker 2 

Marilyn Barman DMH/DD 

Executive Director (671) 
(312-553-9222) 

Peter Bauman 1 

John R. Kroos DMH/DD 

Executive Director (344) 
(815-895-5108) 

Jack McAllister 2 

Vocational Director DMH/DD 

John Winke (576) 
Executive Director 
(312-967-1800) 

Brother Frank Portada 6 
Coord, of Voc. Services DMH/DD 

Jim Weiss (431) 
Executive Director 
(312-424-8616) 

Julie Seta 6 

James OeOre DMH/DD 

Executive Director (407) 
(312-5A3-2440) 

Bob Brocken 2 

Rick Miller DMH/DD 

Executive Director (420) 
(312-863-3803) 

Steve Tenpas 5 

Marjorle Lee DORS 
Presid 

(312-7&. J060) 

Beth Anderson 6 

Peggy White DMH/DD 

Associate Director (459) 
(312-852-7520) 
(312-355-6533) 

Susan Berns Baron 5A 

Ilene Rosenberg DMH/DD 
Dir. of Vocational Serv. (135) 
Carl M. LaMell 
Executive Director 
(312-235-6004) 
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Date/Time 

Staff Facility and Address 



6/29 Warren Achievement Ctr. 
9:00 1314 S. Main St. 
JST Monmouth. IL 61462 



6/23 Cornerstone Services Inc./ 
8:00 Will County Sheltered 
JT Workshop 

2401 W. Jefferson 

Jollet, !L 60435 



Region & 



Contact Person Agency 



Mike Streight I 

Jo McVey OMH/00 

President (428) 
(309-734-8331) 

Cindy Lapicki 6 

Don Hespell DMH/DD 

James Hogan (487) 



Executive Director 
(815-744-7204) 
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Supported Emploi-ment Publications List 



Thank you for expressing interest in Supported Employment. 

The following publications are available from the Illinois Supported 
Employment Program. Please indicate the publications you are interested 
in receiving by entering a (/) next to each article on the enclosed 
order form. Please be sure to total the cost of each publication (current 
costs are figured at St per page), add to this total the cost of mailing, 
and forward a check to: 
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SE1979b 
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Illinois Supported Employment Program 

College rf Education 

University of Illinois 

110 Education Building 
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Champaign, IL 61820 
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mental analysis of a response cost token economy 
in an applied restaurant setting. Scandinavian 
Journal of Behavior Therapy . 6, Supp. 4., 115. 



Rusch, F. R., Connis, R. T., & Sowers, J. (1978). The 
modification and maintenance of time spent attending 
to task using social reinforcement, token rein- 
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setting. Journal of Special Education Te chnology. 
2, 18-26 ^ 

Rusch, F. R. (1979). Toward the validation of social/ 
vocational survival sKills. Mental Retardation, 
17, 143-144, 

Schutz, R. P., Rusch, F. R., & Lawson, D. S. (1979). 
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employment of the mentally retarded adult: Research 
to reality? Journai of Contemporary Business . 8, 
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